acceptodds
Under review as a conference paper at ICLR 2027

Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity

Abstract

Federated fine-tuning of large language models using parameter-efficient methods such as LoRA enables privacy-preserving adaptation of foundation models. Heterogeneous hardware resources introduce challenges, as clients with different adapter ranks cannot be directly aggregated. While existing methods enable aggregation under heterogeneous ranks, they fail to control how information is distributed across rank dimensions, leading to suboptimal use of shared low-rank representations. Instead, we propose PreLort: a nested low-rank formulation for federated LoRA that organizes adapter dimensions into a prefix hierarchy. Our approach ensures that lower-rank dimensions encode task-relevant information, while higher-rank dimensions capture additional capacity. Building on this, we introduce (i) a segment-wise aggregation rule that averages only over clients contributing to each rank segment, avoiding dilution from zero-padded lower-rank clients, and (ii) a prefix-nested training strategy that optimizes each adapter under multiple rank truncations, encouraging useful signal to concentrate in low-rank prefix dimensions. Together, these components encourage a consistent low-rank prefix capturing the most task-relevant information, while higher-rank dimensions learn additional capacity. This allows low-rank clients to benefit from richer information contributed by higher-rank clients, as prefix dimensions are consistently learned and aggregated. Experiments demonstrate that our method consistently outperforms prior heterogeneous federated LoRA methods in accuracy and ROUGE-L, while achieving lower or comparable perplexity across multiple base models.

Primary area:
optimization
Keywords:
federated learning, federated fine-tuning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Nxo7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PreLort targets federated LoRA fine-tuning when clients use different adapter ranks. It has two parts.

  • Nested local training (Eq. 2): each client averages the cross-entropy loss over its own rank and every lower rank level present in the federation, using prefix truncations of the same adapter.
  • Segment-wise aggregation (Eq. 3): the server splits the adapter into rank segments defined by the rank levels. It averages each segment's pseudo-gradients only over the clients whose rank covers that segment, which avoids zero-padding.

The global adapter keeps a fixed rank equal to the largest client rank. Experiments use TinyLlama-1.1B and Qwen2.5-0.5B on Dolly, Alpaca and 20 Newsgroups with 5 or 10 clients. The method is compared with ZeroPad, HetLoRA, FLoRA and FlexLoRA on MMLU accuracy, 20 Newsgroups accuracy, ROUGE-L and perplexity. PreLort has the best MMLU and ROUGE-L values in Table 1. The appendix adds three-seed results, non-IID 20 Newsgroups, a preliminary Qwen2.5-7B run, a FLoRA variant restricted to a per-round effective rank of 64, and wall-clock overhead (about 40%).

Critical strengths
  1. The method is simple and honestly scoped. Eq. 2 and Eq. 3, with Algorithms 1 and 2, describe it in a few lines. Sub-rank passes reuse the adapter through in-place masking, and the paper reports that peak memory does not change. Unlike FLoRA, the global adapter does not grow with the number of clients. The paper also states its limits plainly:

    • It credits averaging over the clients that hold a parameter to FjORD, HeteroFL and FedRolex (Section 2).
    • It notes that cross-client terms BiAjB_i A_j remain (Section 3).
    • It gives no convergence guarantee (Appendix E.1) and makes no privacy claim.
    • It discusses FLoRA's larger per-round rank and adds a rank-64 FLoRA variant (Appendices E.3 and F.2).
    • It measures the extra local compute (Table 9).

    This makes the method easy to adopt and the comparison easy to judge.

  2. Figure 2 shows the intended effect for low-rank clients. In the Alpaca panel, the prefixes of the PreLort adapter keep perplexity near 2.64 from rank 4 to rank 64, close to the homogeneous rank-64 reference. A homogeneous rank-4 model is at about 2.77, and ZeroPad and HetLoRA stay at about 2.80 or above at every rank. A rank-4 client therefore gets a clearly better model than rank-4 training alone would give. The non-IID results add support beyond the IID tables: over three seeds, PreLort improves accuracy over FLoRA restricted to rank 64 by 1.84 points (Qwen2.5-0.5B) and 3.28 points (TinyLlama-1.1B) (Table 5). This is the main evidence that the method is useful in practice.

Critical weaknesses
  1. Most of the reported advantage is within run-to-run variation, and the main accuracy metric is not clearly defined.

    • Margins are small. Table 1 is a single run (Appendix F.1). Except for TinyLlama-1.1B/Dolly MMLU (+3.96 over HetLoRA), the margins over the best external baseline are 0.97 to 1.57 points on MMLU and 0.27 to 0.81 points on ROUGE-L. On TinyLlama-1.1B/20 Newsgroups, FLoRA is higher (34.08 vs. 32.47).
    • They shrink or change across seeds.
      • The only three-seed IID check (Table 4) reduces PreLort's lead over FLoRA from 1.01 points in Table 1 to 0.31 points (39.15 ± 0.58 vs. 38.84 ± 0.50).
      • In the single-seed non-IID run with all baselines (Table 6), unrestricted FLoRA is first, and PreLort is 0.45 points above FlexLoRA.
      • On Qwen2.5-7B, the single-seed run (Table 7) gives 54.0 for PreLort and 53.3 for FlexLoRA, while the three-seed means (Table 8) are 50.51 ± 2.49 and 45.46 ± 2.32. With three runs, no single run can lie 7.8 points above a mean whose standard deviation is 2.32, so the Table 7 run is not one of the three seeds. The 7B gap therefore ranges from 0.7 to about 5 points between runs.
    • The metric is unclear. Section 5.1 reports MMLU accuracy after fine-tuning on Dolly and Alpaca, while Appendix B says accuracy is measured on each dataset's held-out test set. Neither the MMLU protocol nor a base-model score is given. Without a base-model row, a higher MMLU score cannot be attributed to better adaptation rather than to less forgetting or to answer-format effects.

    These points weaken the abstract's claim that PreLort "consistently outperforms" prior methods.

  2. The step beyond FjORD and HeteroFL rests on a misalignment argument that the paper does not test.

    • Both ingredients already exist. FjORD trains ordered prefixes and, like HeteroFL, averages each parameter over the clients that hold it (Section 2, Appendix E.1). The paper separates its contribution by noting that LoRA factorizations are not identifiable, BA=(BQ)(Q−1A)BA = (BQ)(Q^{-1}A).
    • The non-identifiability argument is weak. Every client starts each round from the current global adapter (Section 3), which anchors the coordinates. Since the loss depends on BB and AA only through BABA, gradient steps do not move along this symmetry, so clients diverge because their data differ. This is the same situation as the channels in FjORD and HeteroFL, whose order is also defined only up to permutation. The paper's own explanation of why Nested Aggregation Only fails (Section 5.3.2) is that high-rank dimensions are "supported by only a subset of clients", which is the ordered-dropout rationale of FjORD rather than an identifiability problem.
    • Misalignment is never measured. Representation misalignment across rank dimensions is the first listed contribution, yet the paper reports no direct measurement of it. Figure 3 shows importance concentrating in the leading dimensions, which Eq. 2 produces by design, because lower indices appear in more loss terms. It does not show that client updates on a shared segment agree more.
    • The ablations are mixed. Nested training does stabilize training: Nested Aggregation Only diverges over rounds in most settings (Figure 4). But on Alpaca (Figure 2), Nested Aggregation alone has prefix perplexities within about 0.01 of PreLort from rank 4 to rank 32. On Qwen2.5-0.5B/20 Newsgroups, Nested Aggregation Only is the best entry in Table 1 (39.93 vs. 39.52).

    This point decides whether the paper offers a new finding about LoRA rank or an application of known techniques to it.

Questions for authors
  1. What is the MMLU protocol (subset, number of shots, prompt, answer extraction), and how does it relate to the held-out test sets described in Appendix B? Please add the base model without federated fine-tuning under the same protocol. Please also clarify the following.

    • Which model is evaluated in Tables 1 and 2: the global rank-64 adapter or the clients' models?
    • How is the LoRA scaling factor set when an adapter is truncated?
    • Is the backbone of Figure 2 Qwen2.5-0.5B? In the Alpaca panel, the end points of ZeroPad, HetLoRA, FLoRA and PreLort match the Qwen2.5-0.5B row of Table 2, but the Nested Training curve ends near 2.67, whereas Table 2 gives 3.4447 for Qwen2.5-0.5B.
    • Do the FlexLoRA perplexities in Table 2 (e.g., 6.7000, 11.0000) come from the same evaluation code as the other methods?

    If the base model scores at or above the fine-tuned models, I would read the MMLU columns as retention rather than adaptation.

  2. Can the authors report three seeds for the other Table 1 settings, in particular TinyLlama-1.1B/Dolly (the largest margin) and the ROUGE-L columns? Can they include unrestricted FLoRA and FlexLoRA in the three-seed non-IID comparison (Table 5)? Please also clarify:

    • How is FLoRA restricted to a per-round effective rank of 64: by client sampling or by truncation?
    • Why do the single-seed 7B results in Table 7 differ so much from the three-seed means in Table 8?
    • Why is TinyLlama-1.1B accuracy on 20 Newsgroups higher under the non-IID partition (35.20 ± 0.60, Table 5) than under the IID partition (32.47, Table 1)?

    Gains that hold across seeds would raise my score.

  3. Can the authors measure cross-client alignment directly, for example the cosine similarity between clients' pseudo-gradients on the shared segment [0,r1)[0, r_1) with and without nested training?

    • A comparison with FjORD-style training that samples one prefix rank per step would show whether training all prefixes is needed.
    • Please report training-only time or FLOPs per method. By Eq. 1, the 10-client setting needs 23 forward and backward passes per batch across clients instead of 10, more than the roughly 40% overhead in Table 9 suggests.
    • Please also give the client participation rate per round.
    • Please discuss DyLoRA (Valipour et al., 2023), which trains LoRA under sampled rank truncation in the centralized setting.

    These answers would change my view of the paper's novelty.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Nxo7 (1/5)

We thank the reviewer for an exceptionally careful reading, for recognizing that the method is simple and honestly scoped, and for highlighting the evidence for low-rank clients in Figure 2 and in the non-IID results. Several of the reviewer's observations identified errors in the paper, which we correct below.

Weakness 1: Most of the reported advantage is within run-to-run variation: Table 1 is a single run, the margins are small, Tables 7 and 8 are inconsistent, and the main accuracy metric is not clearly defined.

We ran three new seeds for TinyLlama-1.1B / Dolly, the setting with the largest margin in Table 1 (accuracy and ROUGE-L), and for TinyLlama-1.1B / 20 Newsgroups, where FLoRA is best in Table 1. Qwen2.5-0.5B / Dolly already has three seeds in Table 4 (Appendix F.1), repeated below together with its ROUGE-L. All three-seed results, including Table 4, are new runs, separate from the single run reported in Table 1. Three-seed runs for the remaining Table 1 settings (Alpaca on both backbones and Qwen2.5-0.5B / 20 Newsgroups) are in progress. We will post them here if they finish before the end of the discussion period, and will include them in the revised paper in any case.

Table R1a: Accuracy (%), mean ±\pm std over three seeds, same configuration as Table 1. The Qwen2.5 / Dolly column is Table 4.

MethodTinyLlama / Dolly (MMLU)Qwen2.5 / Dolly (MMLU)TinyLlama / 20NG
ZeroPad29.70 ±\pm 1.8936.71 ±\pm 2.2516.28 ±\pm 0.78
HetLoRA29.40 ±\pm 0.4138.40 ±\pm 0.5717.55 ±\pm 0.23
FLoRA28.72 ±\pm 0.5338.84 ±\pm 0.5033.85 ±\pm 0.43
FlexLoRA27.28 ±\pm 0.4738.79 ±\pm 0.6131.51 ±\pm 0.79
Nested-Agg-Only29.29 ±\pm 1.6336.95 ±\pm 1.1229.01 ±\pm 1.27
Nested-Training-Only29.03 ±\pm 0.4238.21 ±\pm 0.6431.07 ±\pm 0.31
PreLort33.01 ±\pm 0.6939.15 ±\pm 0.5832.67 ±\pm 0.49

Table R1b: ROUGE-L, mean ±\pm std over three seeds. Qwen2.5 / Dolly uses the same runs as Table 4.

MethodTinyLlama / DollyQwen2.5 / Dolly
ZeroPad27.37 ±\pm 0.2026.42 ±\pm 0.32
HetLoRA28.22 ±\pm 0.1827.59 ±\pm 0.17
FLoRA28.96 ±\pm 0.1928.43 ±\pm 0.24
FlexLoRA28.66 ±\pm 0.2128.76 ±\pm 0.11
Nested-Agg-Only28.30 ±\pm 0.2928.61 ±\pm 0.37
Nested-Training-Only27.01 ±\pm 0.3126.23 ±\pm 0.21
PreLort29.44 ±\pm 0.1228.91 ±\pm 0.16

Over three seeds, PreLort has the highest mean accuracy on TinyLlama-1.1B / Dolly (33.01 ±\pm 0.69 vs. 29.70 ±\pm 1.89 for the best baseline) and the highest ROUGE-L (29.44 ±\pm 0.12 vs. 28.96 ±\pm 0.19 for FLoRA), so this gain holds across seeds. On Qwen2.5-0.5B / Dolly, PreLort also has the highest mean accuracy and ROUGE-L, but both margins (0.31 over FLoRA and 0.15 over FlexLoRA) are within one standard deviation. On TinyLlama-1.1B / 20 Newsgroups, FLoRA remains best (33.85 ±\pm 0.43 vs. 32.67 ±\pm 0.49), as in Table 1. PreLort is above FlexLoRA and above both ablations in every column.

Tables 7 and 8. The reviewer is correct that the single-seed run in Table 7 cannot be one of the three seeds in Table 8. Table 8 pooled runs from two different data partitions, and Appendix F.3 mislabels the setting. We replace both tables with a new experiment on a single partition, with three seeds and 10 communication rounds, on a recent backbone:

Table R5: Qwen3-8B-Base on 20 Newsgroups, 10 clients, ranks [64,32,16,16,8,8,4,4,4,4], 10 communication rounds ×\times 1 local epoch, mean ±\pm std over three seeds.

MethodAccuracy (%)Perplexity
Base model (no fine-tuning)9.0810.2885
ZeroPad30.01 ±\pm 1.568.5413 ±\pm 0.1160
HetLoRA39.73 ±\pm 0.787.3912 ±\pm 0.1059
FLoRA52.92 ±\pm 0.757.3550 ±\pm 0.1134
FlexLoRA63.56 ±\pm 0.457.2726 ±\pm 0.1146
PreLort64.23 ±\pm 0.567.2465 ±\pm 0.1023

On Qwen3-8B-Base with three seeds and 10 rounds, PreLort obtains the highest accuracy (64.23 ±\pm 0.56%), 0.67 points above FlexLoRA and well above FLoRA (+11.3), HetLoRA (+24.5) and ZeroPad (+34.2).

Metric. Section 5.1 is correct: Dolly and Alpaca accuracy is MMLU accuracy after fine-tuning, and Appendix B is wrong. The MMLU protocol and the base-model results are given under Question 1. We will remove "consistently outperforms" from the abstract and report the per-setting findings above.

AuthorsOpenReview ↗

Response to Reviewer Nxo7 (2/5)

Weakness 2: The step beyond FjORD and HeteroFL rests on a misalignment argument that the paper does not test: the non-identifiability argument is weak, misalignment is never measured, and the ablations are mixed.

We respectfully keep the identifiability argument but will sharpen it. A LoRA update is invariant to reordering its rank coordinates (more generally, BA=(BR)(R−1A)BA = (BR)(R^{-1}A) for any invertible RR), so standard training does not tie a direction to a rank index or place the most important directions first. Starting each round from the global adapter fixes the initialization, not this ordering; FjORD faces the same situation, since its clients also start from the global model every round, and resolves it with ordered dropout. Table R4 confirms that anchoring alone is not sufficient: with the same initialization and the same segment-wise aggregation, client agreement on the shared segment is 0.300 without nested training vs. 0.633 with it (Qwen2.5-0.5B / 20 Newsgroups). Existing rank-heterogeneous LoRA methods (HetLoRA, FlexLoRA, FLoRA, zero-padding) do not address this ambiguity. To our knowledge, we are the first to identify it in this setting and to resolve it with FjORD-style nestedness: (i) rank index ii receives supervision from every prefix objective whose width includes ii (Eq. 2), which orders the coordinates, and (ii) segment ll is averaged only over clients with rk≥rlr_k \geq r_l. We will rewrite Sections 2 and 3 in these terms.

Alignment measured directly. Figure 3 shows a concentration of importance, which Eq. 2 produces by design; it does not show that client updates agree. We therefore measured the cosine similarity between clients' round updates on the shared segment [0,r1)[0, r_1), with and without nested training, under the same segment-wise aggregation. We use the dense update ΔW\Delta W of the prefix, which does not depend on how each client parameterizes its factors:

Table R4: Mean pairwise cosine similarity between clients' round updates ΔW\Delta W on the shared segment [0,r1)[0, r_1), rounds ≥1\geq 1, seed 42.

SettingPreLortNested-Agg-Only (no nested training)Sampled prefixZeroPadFLoRAFlexLoRA
Qwen2.5-0.5B / 20 Newsgroups (10 clients, r1=4r_1=4)0.6330.3000.7180.1660.0750.250
TinyLlama-1.1B / Dolly (5 clients, r1=1r_1=1)0.5130.1590.5400.0140.0020.343
Qwen3-8B-Base / 20 Newsgroups (10 clients, r1=4r_1=4)0.5560.2390.5930.1270.0800.230

Under the same segment-wise aggregation, nested training raises client agreement on the shared segment by about 2-3 ×\times (0.300 →\rightarrow 0.633, 0.159 →\rightarrow 0.513, 0.239 →\rightarrow 0.556), well above ZeroPad and FLoRA (≤0.17\leq 0.17); the sampled prefix gives similar agreement, so the alignment comes from nested training itself.

All prefixes vs. one sampled prefix per step: see Table R3 under Question 3.

Ablations. We agree that they are mixed in Table 1: Nested-Agg-Only is the best entry on Qwen2.5-0.5B / 20 Newsgroups, and on Alpaca its prefix perplexities are within about 0.01 of PreLort from rank 4 to rank 32 (Fig. 2). Over three seeds (Tables R1a and R1b), PreLort is above both Nested-Agg-Only and Nested-Training-Only in every setting we reran; the three-seed Qwen2.5-0.5B / 20 Newsgroups comparison is in progress (see Weakness 1). We will describe Nested-Agg-Only as unstable over rounds (Fig. 4) but competitive in some settings, rather than as uniformly inferior.

AuthorsOpenReview ↗

Response to Reviewer Nxo7 (3/5)

Question 1: What is the MMLU protocol, and how does it relate to the held-out test sets in Appendix B? Please add the base model under the same protocol, and clarify which model is evaluated, the LoRA scaling under truncation, the backbone of Figure 2, and the FlexLoRA perplexities in Table 2.

MMLU protocol. Accuracy is measured on MMLU, as stated in Section 5.1; Appendix B incorrectly says each dataset's held-out test set, and we will correct it. Models are fine-tuned on Dolly or Alpaca and then evaluated zero-shot, without any MMLU training data, on a fixed 1,444-question subset of the MMLU test set covering all 57 subjects (the subset used in FLoRA's evaluation code). The model generates a short answer, which is counted as correct if it contains the gold option, and we report the mean of per-subject accuracies. The Dolly and Alpaca test sets are used only for ROUGE-L and perplexity.

Base model. Under the same protocol and evaluation code:

Table R2: Base models without federated fine-tuning, same evaluation code as Table 1.

BackboneMMLU (%)20 Newsgroups Acc. (%)ROUGE-L DollyROUGE-L Alpaca
TinyLlama-1.1B (base)33.210.7423.3323.90
Qwen2.5-0.5B (base)35.350.5424.5027.03

Fine-tuning improves the task metrics (PreLort: ROUGE-L +4.3 to +9.9; 20 Newsgroups accuracy from <1% to 32.5–39.5%), while MMLU stays within a few points of the base model; we agree that MMLU here measures retention, and will present it as such.

Which model is evaluated. The server's global model after the final round, never the clients' local models. For PreLort, this is the global adapter at the maximum client rank on the frozen backbone; for FLoRA, it is the backbone with the stacked updates merged.

LoRA scaling under truncation. All adapters use α=2r\alpha = 2r, so the scaling factor α/r=2\alpha / r = 2 is the same for every client and for the global adapter. Truncation, both in Figure 2 and in the sub-rank passes of Eq. 2, zeroes the tail rank dimensions in place, so the scaling stays at 2 for every prefix.

Figure 2. The backbone is Qwen2.5-0.5B. The reviewer is right that the Nested Training curve in the Alpaca panel does not match Table 2: its end point (about 2.67) is the TinyLlama-1.1B value (2.6699), so this curve was plotted from the wrong run. We will regenerate the figure from the Qwen2.5-0.5B run.

FlexLoRA perplexities. They come from the same evaluation code as the other methods; the FlexLoRA entries in Table 2 were transcribed with two decimals. We will report the full-precision values.

Reviewer_JYtxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors target the problem of heterogeneous federated parameter-efficient fine-tuning (LoRA). In LoRA, instead of updating the whole weights, for each layer we train two low-rank matrices. Since in FL we have clients with different computational power, it is reasonable to assign different ranks to the LoRA modules for different clients accordingly. The authors first state that blindly aggregating such updates with different ranks (e.g., via zero-padding) can dilute the updates or cause misalignment across clients’ representations. Therefore, they propose PreLort, which is a combination of two components: 1) Nested Local Training, where a client with assigned rank rir_i also evaluates its adapter across all lower client ranks in a Matryoshka-style training objective, and 2) Segment-Wise Aggregation, where during server aggregation, for each rank segment, they only average over the clients that participated in training and updating that segment. As empirical evaluations, the authors evaluate their method on several LLMs across different datasets and settings.

Critical strengths
  • The empirical evaluations are conducted on recent and relatively large-scale language models.

  • The authors have done a proper investigation in their ablation studies. Particularly, they show that nested local training plays an important role in keeping representation importance concentrated in the leading dimensions.

  • The final global model preserves its performance when truncated to different ranks, showing that the learned hierarchy is beneficial across clients of varying capacities.

Critical weaknesses

My primary critique of the paper is that all of its core algorithmic components have already been introduced in prior work and nothing special is designed that is tailored for FL setting.

  • Prefix-nested LoRA has already been proposed in DyLoRA (Valipour et al., EACL 2023), which similarly truncates LoRA matrices to prefix ranks during training to order rank importance.

  • Segment-wise aggregation is a standard approach in heterogeneous FL settings (e.g., HeteroFL, FjORD).

Overall, the paper reuses existing LoRA and heterogeneous FL methodology. Authors' diagnosis of why naive partial aggregation fails in heterogeneous FL and the fix have already been explored in many prior FL works.

Beyond novelty, I have several concerns regarding the experimental setup and claims:

  • The proposed algorithm introduces additional computation costs on the client side and it is not clear that the authors adjusted the training budget of the baselines to ensure a fair, compute-matched comparison.

  • The main setup relies on a small number of clients (e.g., 10 clients in the primary setting) participating every round. In realistic cross-device FL, clients frequently drop out or fail to share updates (for example the client with the highest rank might miss the deadline and fail to participate). The paper does not study or address this cross-segment staleness issue.

  • The claim in the text that PreLort "consistently achieves the best performance" is an overstatement, as Table 1 shows that baseline methods achieve better performance on 20 Newsgroups. Also, in several settings the gap between different methods is very negligible and can be due to random seeding.

Questions for authors

Please check out the weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer JYtx (1/2)

We thank the reviewer for the feedback, and for recognizing our ablation study, in particular the role of nested local training in concentrating importance in the leading dimensions, and the robustness of the global model under truncation. We address the raised concerns below.

Weakness 1: All core components exist in prior work: prefix-nested LoRA in DyLoRA, and segment-wise aggregation in HeteroFL and FjORD. Nothing is specifically designed for the federated setting.

We agree that prefix truncation of LoRA (DyLoRA [Valipour et al., 2023]) and averaging each parameter over the clients that hold it (HeteroFL, FjORD) are known. We will add DyLoRA to the related work. What is specific to our setting is the problem these components solve: a LoRA update is invariant to reordering its rank coordinates, so nothing ties the most important directions to the prefix that low-rank clients keep. FjORD resolves the same identifiability problem for widths, and existing heterogeneous LoRA methods do not address it (see our response to Reviewer Nxo7, Weakness 2).

The design choices specific to the federated setting are the following:

  • The rank levels, and therefore the segments, are defined by the federation's client ranks, not by a fixed or sampled rank schedule.
  • The server keeps one adapter bounded by the maximum client rank; its prefixes serve every client's budget, and it does not grow with the number of clients (unlike FLoRA).

A concrete difference from DyLoRA and FjORD is that they train one sampled rank per step, whereas PreLort trains all federation rank levels up to the client's rank at every step (Eq. 2). To test whether this matters, we ran the sampled variant inside the same pipeline, with the same segment-wise aggregation and sorting: at every optimizer step, each client draws one prefix rank uniformly from its admissible rank levels.

Table R3: All prefixes (PreLort) vs. one prefix rank drawn uniformly at random per optimizer step from the client's admissible levels (FjORD/DyLoRA-style), same aggregation and sorting, seed 42. PreLort entries are the Table 1/2 runs; the sampled-prefix runs use the same configuration.

SettingMetricPreLort (all prefixes)Sampled prefixTraining time, sampled / PreLort
TinyLlama-1.1B / DollyROUGE-L ↑\uparrow29.3426.400.55 ×\times
TinyLlama-1.1B / DollyPerplexity ↓\downarrow4.40304.4246
Qwen2.5-0.5B / 20 NewsgroupsAccuracy (%) ↑\uparrow39.5229.100.71 ×\times
TinyLlama-1.1B / 20 NewsgroupsAccuracy (%) ↑\uparrow32.4726.630.56 ×\times

The ablations in the paper further show that the coupling is necessary: segment-wise aggregation alone diverges over rounds (Fig. 4), and nested training alone is weaker (Tables 1-2).

Weakness 2: Nested training adds client-side computation, and it is not clear that the baselines received a compute-matched training budget.

All methods use the same number of rounds, local epochs, batch size and learning rate, so the baselines were not given additional local steps. The sampled-prefix variant above (Table R3) performs one forward and backward pass per batch, exactly like every baseline, so it is the compute-matched version of our method. We also measured the training-only time of every method:

Table R6: Training-only time (sum of the clients' local training time over all rounds, excluding evaluation), averaged over completed seeds. Baselines are ZeroPad, HetLoRA, FLoRA, and FlexLoRA (HetLoRA, FLoRA, and FlexLoRA for Qwen3-8B-Base).

SettingPasses per batch across clients (PreLort / baselines)Baselines, mean (h)PreLort (h)PreLort / baselinesSampled prefix / baselines
TinyLlama-1.1B / Dolly (5 clients)15 / 55.297.921.50 ×\times0.84 ×\times
Qwen2.5-0.5B / 20 Newsgroups (10 clients)23 / 100.741.121.51 ×\times1.07 ×\times
Qwen3-8B-Base / 20 Newsgroups (10 clients)23 / 103.214.631.44 ×\times0.96 ×\times

The training-only overhead of PreLort is 1.4-1.5 ×\times that of the baselines, below the 2.3-3 ×\times increase in forward and backward passes per batch, while the sampled-prefix variant costs 0.84-1.07 ×\times. Table 9 in the paper reports end-to-end time per run, which includes evaluation and therefore understates this overhead. We will report both quantities.

AuthorsOpenReview ↗

Response to Reviewer JYtx (2/2)

Weakness 3: The setup uses a small number of clients, all of whom participate in every round. Client dropout and cross-segment staleness (e.g., when the highest-rank client misses a round) are not studied.

We confirm that all clients participate in every round. Clients are sampled uniformly at random with a selection fraction of 1.0, which Section 5.1 does not state; we will add this. We agree that the highest-rank client missing a round is the critical case, because the top segment then has no contributor in that round. Our current implementation forms segments from the participating clients' ranks, so the global adapter would temporarily lose its top segment. The fix that is consistent with Eq. 3 is to keep the previous value of any segment without a contributor. We will implement this rule in the released code.

In scope, our study targets cross-silo federated learning with a small number of clients. We will state this explicitly.

Weakness 4: The claim that PreLort "consistently achieves the best performance" is an overstatement: baselines are better on 20 Newsgroups, and several gaps may be due to random seeds.

We agree and will revise the abstract, Section 5.2 and the conclusion. We ran three seeds for TinyLlama-1.1B / Dolly and TinyLlama-1.1B / 20 Newsgroups, alongside the three-seed Qwen2.5-0.5B / Dolly runs of Table 4 (Tables R1a and R1b, in our response to Reviewer Nxo7, Weakness 1); the remaining settings are in progress.

We will report these per-setting findings directly rather than claim consistent superiority.

We thank the reviewer for their detailed comments and would appreciate their acknowledgment if our rebuttal has addressed their concerns. If any parts of the rebuttal are unclear or require further experimentation, we would be happy to address them during the discussion period.

Reviewer_gEqbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses rank heterogeneity in federated LoRA. It argues that existing methods (zero-padding, HetLoRA, FLoRA, FlexLoRA) focus on combining updates algebraically but ignore that rank coordinates play different roles across clients. PreLort couples two components. First, all-prefix nested local training averages the loss over every system rank level < r_k, so each prefix of an adapter is usable on its own. Second, segment-wise aggregation averages each rank segment’s pseudo-gradients only over the clients that hold it, which avoids zero-padding dilution. On TinyLlama-1.1B and Qwen2.5-0.5B with Dolly, Alpaca, and 20 Newsgroups, PreLort reports the best accuracy and ROUGE-L in most settings and competitive perplexity. Ablations show that neither component suffices alone, and the trained adapter remains robust under prefix truncation, at about 40% extra local compute.

Critical strengths
  1. The training–aggregation coupling is well supported by the ablations. Aggregation-only diverges in perplexity (Fig. 4) and lacks leading-dimension concentration (Fig. 3). The full method is stable and robust under prefix truncation (Fig. 2). This shows the contribution lies in the combination rather than in either component alone.

  2. The method is practical. It adds no communication or peak-memory cost, keeps the adapter bounded by the maximum client rank, and yields a single adapter that can be truncated to any client’s budget.

Critical weaknesses
  1. Novelty is limited, and the motivating mechanism is untested. Nested prefix training (FjORD, slimmable networks) and support-aware averaging (HeteroFL, FedRolex) are both established. The claimed “rank-coordinate misalignment” is never measured directly.

  2. The empirical evaluation is dated and statistically weak. The backbones (TinyLlama-1.1B, Qwen2.5-0.5B) and benchmarks (Alpaca, Dolly, 20 Newsgroups, MMLU) are old and small-scale. The 7B result runs for only two rounds.

Questions for authors
  1. Is the Dolly/Alpaca accuracy in Table 1 measured on MMLU or on the held-out test sets? How is it evaluated?

  2. Can you add experiments with recent backbones (e.g., Llama-3 or Qwen3 at 7B+) and benchmarks?

  3. How exactly is FLoRA restricted to a per-round effective rank of 64: by truncating client ranks, subsampling clients, or SVD truncation of the stacked update?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gEqb (1/2)

We thank the reviewer for the detailed questions and for recognizing that the coupling of nested training and segment-wise aggregation is supported by the ablations and that the method is practical. We address each concern raised below.

Weakness 1: Novelty is limited, and the motivating mechanism is untested. Nested prefix training (FjORD, slimmable networks) and support-aware averaging (HeteroFL, FedRolex) are both established. The claimed “rank-coordinate misalignment” is never measured directly.

We agree that both ingredients exist individually, and we will state this explicitly. Ordered prefix training appears in FjORD, slimmable networks, and DyLoRA [Valipour et al., 2023], and averaging each parameter over the clients that hold it appears in HeteroFL, FjORD, and FedRolex. Our contribution is to identify that rank-heterogeneous federated LoRA has the identifiability problem that FjORD addresses for widths: a LoRA update is invariant to rotation of its rank coordinates, so nothing ties the most important directions to the prefix that low-rank clients keep. Existing heterogeneous LoRA methods do not address this; to our knowledge, we are the first to draw this connection and to resolve it by coupling FjORD-style nested training with segment-wise aggregation, and neither component works alone: segment-wise aggregation without nested training diverges over rounds (Fig. 4) in most settings, and nested training with zero-padding aggregation is weaker (Tables 1-2). Together, they produce a single global adapter that every client can truncate to its own budget (Fig. 2).

We now measure cross-client alignment directly. In every round, we compute the cosine similarity between clients' updates on the rank segment shared by all clients, [0,r1)[0, r_1). We use the dense update ΔW\Delta W of that prefix, which does not depend on how each client parameterizes its factors, so the measure is comparable across methods:

Table R4: Mean pairwise cosine similarity between clients' round updates ΔW\Delta W on the shared segment [0,r1)[0, r_1), rounds ≥1\geq 1, seed 42 ↑\uparrow

SettingPreLortNested-Agg-Only (no nested training)Sampled prefixZeroPadFLoRAFlexLoRA
Qwen2.5-0.5B / 20 Newsgroups (10 clients, r1=4r_1=4)0.6330.3000.7180.1660.0750.250
TinyLlama-1.1B / Dolly (5 clients, r1=1r_1=1)0.5130.1590.5400.0140.0020.343
Qwen3-8B-Base / 20 Newsgroups (10 clients, r1=4r_1=4)0.5560.2390.5930.1270.0800.230

Under the same segment-wise aggregation, nested training raises the mean agreement of client updates on the shared segment from 0.300 (Nested-Agg-Only) to 0.633 (PreLort) on Qwen2.5-0.5B / 20 Newsgroups, from 0.159 to 0.513 on TinyLlama-1.1B / Dolly, and from 0.239 to 0.556 on Qwen3-8B-Base. PreLort's agreement is 1.5-2.5 ×\times that of FlexLoRA and far above ZeroPad and FLoRA (at most 0.17) in all three settings. Training one sampled prefix per step gives similar or slightly higher agreement (0.718, 0.540 and 0.593), so the alignment comes from nested training itself. Higher agreement does not by itself imply higher full-rank accuracy: Nested-Agg-Only is more accurate than PreLort on Qwen2.5-0.5B / 20 Newsgroups (Table 1).

Weakness 2: The empirical evaluation is dated and statistically weak. The backbones (TinyLlama-1.1B, Qwen2.5-0.5B) and benchmarks (Alpaca, Dolly, 20 Newsgroups, MMLU) are old and small-scale. The 7B result runs for only two rounds.

We added a recent backbone, Qwen3-8B-Base, trained for 10 communication rounds instead of 2, on 20 Newsgroups with 10 clients, using the paper's rank configuration [64,32,16,16,8,8,4,4,4,4] (Table R5, under Question 2). We keep Dolly, Alpaca, and 20 Newsgroups with MMLU so that our results stay comparable with the FLoRA protocol, whose evaluation code we reuse. We agree that newer benchmarks would strengthen the evaluation and will add one in the revision.

AuthorsOpenReview ↗

Response to Reviewer gEqb (2/2)

Question 1: Is the Dolly/Alpaca accuracy in Table 1 measured on MMLU or on the held-out test sets? How is it evaluated?

It is measured on MMLU, as stated in Section 5.1. Appendix B incorrectly says "each dataset’s held-out test set," and we will correct it. Models are fine-tuned on Dolly or Alpaca and then evaluated zero-shot, without any MMLU training data, on a fixed 1,444-question subset of the MMLU test set covering all 57 subjects (the subset used in FLoRA’s evaluation code). The model generates a short answer, which is counted as correct if it contains the gold option, and we report the mean of per-subject accuracies. The Dolly and Alpaca test sets are used only for ROUGE-L and perplexity.

Under the same protocol, the base models without federated fine-tuning score:

Table R2: Base models without federated fine-tuning, same evaluation code as Table 1.

BackboneMMLU (%)20 Newsgroups Acc. (%)ROUGE-L DollyROUGE-L Alpaca
TinyLlama-1.1B (base)33.210.7423.3323.90
Qwen2.5-0.5B (base)35.350.5424.5027.03

Fine-tuning yields large gains on the task metrics: relative to the base model, PreLort improves ROUGE-L by 4.3-9.9 points and raises 20 Newsgroups accuracy from below 1% to 32.5% (TinyLlama-1.1B) and 39.5% (Qwen2.5-0.5B); the base models rarely output a valid newsgroup label. MMLU, in contrast, moves only a few points from the base model in either direction (PreLort: −2.5 to +4.5; baselines: −6.5 to +3.5), because Dolly and Alpaca teach instruction following rather than new knowledge. The MMLU columns therefore measure how much general knowledge is retained after federated fine-tuning. We will state this in Section 5.1 and lead with ROUGE-L, perplexity, and 20 Newsgroups accuracy.

Question 2: Can you add experiments with recent backbones (e.g., Llama-3 or Qwen3 at 7B+)?

Yes. We ran Qwen3-8B-Base on 20 Newsgroups with 10 clients and ranks [64,32,16,16,8,8,4,4,4,4], for 10 communication rounds with one local epoch per round:

Table R5: Qwen3-8B-Base on 20 Newsgroups, 10 clients, ranks [64,32,16,16,8,8,4,4,4,4], 10 communication rounds ×\times 1 local epoch, mean ±\pm std over three seeds.

MethodAccuracy (%) ↑\uparrowPerplexity ↓\downarrow
Base model (no fine-tuning)9.0810.2885
ZeroPad30.01 ±\pm 1.568.5413 ±\pm 0.1160
HetLoRA39.73 ±\pm 0.787.3912 ±\pm 0.1059
FLoRA52.92 ±\pm 0.757.3550 ±\pm 0.1134
FlexLoRA63.56 ±\pm 0.457.2726 ±\pm 0.1146
PreLort64.23 ±\pm 0.567.2465 ±\pm 0.1023

On a recent 8B backbone with 10 communication rounds, PreLort obtains the highest mean accuracy (64.23 ±\pm 0.56%), 0.67 points above FlexLoRA (about one standard deviation) and well above FLoRA (+11.3 points), HetLoRA (+24.5), and ZeroPad (+34.2).

Question 3: How exactly is FLoRA restricted to a per-round effective rank of 64: by truncating client ranks, subsampling clients, or SVD truncation of the stacked update?

By truncating client ranks. The ten clients use ranks [16,16,8,8,4,4,4,2,1,1] instead of [64,32,16,16,8,8,4,4,4,4], so the stacked per-round update has rank ∑krk=64\sum_k r_k = 64, equal to the rank of PreLort's persistent global adapter. We do not subsample clients or apply SVD truncation to the stacked update. We will state this explicitly in Appendix F.2.

We thank the reviewer for their detailed comments and would appreciate their acknowledgment if our rebuttal has addressed their concerns. If any parts of the rebuttal are unclear or require further experimentation, we would be happy to address them during the discussion period.

Reviewer_gEqbOpenReview ↗

Reply to authors

Thank you for the detailed response and the additional experiments. The clarifications on the MMLU protocol and the FLoRA restriction are helpful, and the Qwen3-8B run with 10 rounds and three seeds is a clear improvement over the two-round 7B result. Several concerns remain:

  1. The sampled-prefix variant achieves higher alignment than PreLort in every setting, so please report its accuracy, since otherwise the ~40% extra compute of all-prefix training is not justified and the Nested-Agg-Only result already shows that alignment does not track accuracy.

  2. The base scores in Table R2 are inconsistent with public numbers (TinyLlama-1.1B ≈ 26% and Qwen2.5-0.5B 47.5% on standard MMLU vs. 33.21% and 35.35% here), and 13 of 14 TinyLlama entries and 6 of 7 Qwen/Alpaca entries fall below the base model, so the MMLU columns mainly reflect a noisy, format-sensitive measure of forgetting rather than a gain from PreLort.

  3. On Qwen3-8B, PreLort’s margin over FlexLoRA (64.23±0.56 vs. 63.56±0.45) is within one standard deviation and shrinks from the 5-point gap seen after two rounds on Qwen2.5-7B, which suggests the advantage fades as training converges.

Overall, the rebuttal improves clarity, but the novelty concern and the statistical strength of the gains remain largely unchanged, so I keep my original score.

AuthorsOpenReview ↗

Reply to Reviewer gEqb

Thank you for the follow-up.

1. Accuracy of the sampled-prefix variant. We reported it in Table R3, in our responses to Reviewer JYtx (Weakness 1) and Reviewer Nxo7 (Question 3), but did not repeat it here. We apologize for the omission.

SettingMetricPreLort (all prefixes)Sampled prefixTraining time, sampled / PreLort
TinyLlama-1.1B / DollyROUGE-L ↑\uparrow29.3426.400.55 ×\times
TinyLlama-1.1B / DollyPerplexity ↓\downarrow4.40304.4246
Qwen2.5-0.5B / 20 NewsgroupsAccuracy (%) ↑\uparrow39.5229.100.71 ×\times
TinyLlama-1.1B / 20 NewsgroupsAccuracy (%) ↑\uparrow32.4726.630.56 ×\times

Training all prefixes is better in every setting, even though the sampled variant has slightly higher agreement (Table R4). Alignment shows that nested training makes the shared coordinates agree across clients, which is what segment-wise aggregation needs; it is necessary but not sufficient for accuracy. Given aligned coordinates, accuracy also depends on how well each prefix is trained: PreLort optimizes every rank level at every step, whereas the sampled variant optimizes each prefix only on the steps where it is drawn. This is also the comparison that justifies the extra compute: all-prefix training takes 1.4-1.8 ×\times the training time of the sampled variant, and 1.4-1.5 ×\times that of the baselines (Table R6, in our response to Reviewer JYtx, Weakness 2).

2. MMLU. The public numbers come from a different protocol: 5-shot log-likelihood scoring over the full MMLU test set. Table 1 follows FLoRA's evaluation code: zero-shot, generated answers, a fixed 1,444-question subset, an answer counted as correct if the generation contains the gold option, and per-subject accuracies averaged. The two protocols differ in prompt, scoring and question set, so their absolute scores are not comparable. We agree with the reviewer's reading, as stated in our response: after instruction fine-tuning on Dolly and Alpaca, MMLU here measures how much general knowledge is retained, not a gain. We do not claim an MMLU gain from PreLort. Our claims rest on ROUGE-L, perplexity and 20 Newsgroups accuracy, where fine-tuning clearly helps (Table R2). In the revision, we will present MMLU as a secondary retention metric.

3. Qwen3-8B. We agree that the 0.67-point margin over FlexLoRA is within one standard deviation. We said so in our response, and we do not claim a significant gain over FlexLoRA on this backbone. The two-round Qwen2.5-7B result was a single seed with a different backbone and training length, so the change in the gap cannot be attributed to convergence alone. We will report the three-seed 8B result as on par with FlexLoRA and clearly above FLoRA, HetLoRA and ZeroPad (+11.3 to +34.2 points).

Novelty. Table R3 isolates what PreLort adds beyond DyLoRA and FjORD. Under identical aggregation and sorting, training all federation rank levels jointly at each step, rather than one sampled rank, is better in every setting we tested.

We thank the reviewer again for the careful reading and the suggestions. We are happy to address any further questions.

Comments on the paper

1 comment

Dear Reviewers,

Thank you to those who have already contributed to the discussion.

As the discussion period draws to a close, I’d especially appreciate hearing from Reviewers Reviewer gEqb, Reviewer JYtx, and Reviewer Nxo7, who have yet to join the discussion. Please post a brief reply to the authors indicating which concerns their response has addressed and which, if any, remain. If all your concerns have been resolved, a short confirmation is sufficient.

Thank you for your time and input.

Best,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.