- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PreLort targets federated LoRA fine-tuning when clients use different adapter ranks. It has two parts.
- Nested local training (Eq. 2): each client averages the cross-entropy loss over its own rank and every lower rank level present in the federation, using prefix truncations of the same adapter.
- Segment-wise aggregation (Eq. 3): the server splits the adapter into rank segments defined by the rank levels. It averages each segment's pseudo-gradients only over the clients whose rank covers that segment, which avoids zero-padding.
The global adapter keeps a fixed rank equal to the largest client rank. Experiments use TinyLlama-1.1B and Qwen2.5-0.5B on Dolly, Alpaca and 20 Newsgroups with 5 or 10 clients. The method is compared with ZeroPad, HetLoRA, FLoRA and FlexLoRA on MMLU accuracy, 20 Newsgroups accuracy, ROUGE-L and perplexity. PreLort has the best MMLU and ROUGE-L values in Table 1. The appendix adds three-seed results, non-IID 20 Newsgroups, a preliminary Qwen2.5-7B run, a FLoRA variant restricted to a per-round effective rank of 64, and wall-clock overhead (about 40%).
Critical strengths
-
The method is simple and honestly scoped. Eq. 2 and Eq. 3, with Algorithms 1 and 2, describe it in a few lines. Sub-rank passes reuse the adapter through in-place masking, and the paper reports that peak memory does not change. Unlike FLoRA, the global adapter does not grow with the number of clients. The paper also states its limits plainly:
- It credits averaging over the clients that hold a parameter to FjORD, HeteroFL and FedRolex (Section 2).
- It notes that cross-client terms remain (Section 3).
- It gives no convergence guarantee (Appendix E.1) and makes no privacy claim.
- It discusses FLoRA's larger per-round rank and adds a rank-64 FLoRA variant (Appendices E.3 and F.2).
- It measures the extra local compute (Table 9).
This makes the method easy to adopt and the comparison easy to judge.
-
Figure 2 shows the intended effect for low-rank clients. In the Alpaca panel, the prefixes of the PreLort adapter keep perplexity near 2.64 from rank 4 to rank 64, close to the homogeneous rank-64 reference. A homogeneous rank-4 model is at about 2.77, and ZeroPad and HetLoRA stay at about 2.80 or above at every rank. A rank-4 client therefore gets a clearly better model than rank-4 training alone would give. The non-IID results add support beyond the IID tables: over three seeds, PreLort improves accuracy over FLoRA restricted to rank 64 by 1.84 points (Qwen2.5-0.5B) and 3.28 points (TinyLlama-1.1B) (Table 5). This is the main evidence that the method is useful in practice.
Critical weaknesses
-
Most of the reported advantage is within run-to-run variation, and the main accuracy metric is not clearly defined.
- Margins are small. Table 1 is a single run (Appendix F.1). Except for TinyLlama-1.1B/Dolly MMLU (+3.96 over HetLoRA), the margins over the best external baseline are 0.97 to 1.57 points on MMLU and 0.27 to 0.81 points on ROUGE-L. On TinyLlama-1.1B/20 Newsgroups, FLoRA is higher (34.08 vs. 32.47).
- They shrink or change across seeds.
- The only three-seed IID check (Table 4) reduces PreLort's lead over FLoRA from 1.01 points in Table 1 to 0.31 points (39.15 ± 0.58 vs. 38.84 ± 0.50).
- In the single-seed non-IID run with all baselines (Table 6), unrestricted FLoRA is first, and PreLort is 0.45 points above FlexLoRA.
- On Qwen2.5-7B, the single-seed run (Table 7) gives 54.0 for PreLort and 53.3 for FlexLoRA, while the three-seed means (Table 8) are 50.51 ± 2.49 and 45.46 ± 2.32. With three runs, no single run can lie 7.8 points above a mean whose standard deviation is 2.32, so the Table 7 run is not one of the three seeds. The 7B gap therefore ranges from 0.7 to about 5 points between runs.
- The metric is unclear. Section 5.1 reports MMLU accuracy after fine-tuning on Dolly and Alpaca, while Appendix B says accuracy is measured on each dataset's held-out test set. Neither the MMLU protocol nor a base-model score is given. Without a base-model row, a higher MMLU score cannot be attributed to better adaptation rather than to less forgetting or to answer-format effects.
These points weaken the abstract's claim that PreLort "consistently outperforms" prior methods.
-
The step beyond FjORD and HeteroFL rests on a misalignment argument that the paper does not test.
- Both ingredients already exist. FjORD trains ordered prefixes and, like HeteroFL, averages each parameter over the clients that hold it (Section 2, Appendix E.1). The paper separates its contribution by noting that LoRA factorizations are not identifiable, .
- The non-identifiability argument is weak. Every client starts each round from the current global adapter (Section 3), which anchors the coordinates. Since the loss depends on and only through , gradient steps do not move along this symmetry, so clients diverge because their data differ. This is the same situation as the channels in FjORD and HeteroFL, whose order is also defined only up to permutation. The paper's own explanation of why Nested Aggregation Only fails (Section 5.3.2) is that high-rank dimensions are "supported by only a subset of clients", which is the ordered-dropout rationale of FjORD rather than an identifiability problem.
- Misalignment is never measured. Representation misalignment across rank dimensions is the first listed contribution, yet the paper reports no direct measurement of it. Figure 3 shows importance concentrating in the leading dimensions, which Eq. 2 produces by design, because lower indices appear in more loss terms. It does not show that client updates on a shared segment agree more.
- The ablations are mixed. Nested training does stabilize training: Nested Aggregation Only diverges over rounds in most settings (Figure 4). But on Alpaca (Figure 2), Nested Aggregation alone has prefix perplexities within about 0.01 of PreLort from rank 4 to rank 32. On Qwen2.5-0.5B/20 Newsgroups, Nested Aggregation Only is the best entry in Table 1 (39.93 vs. 39.52).
This point decides whether the paper offers a new finding about LoRA rank or an application of known techniques to it.
Questions for authors
-
What is the MMLU protocol (subset, number of shots, prompt, answer extraction), and how does it relate to the held-out test sets described in Appendix B? Please add the base model without federated fine-tuning under the same protocol. Please also clarify the following.
- Which model is evaluated in Tables 1 and 2: the global rank-64 adapter or the clients' models?
- How is the LoRA scaling factor set when an adapter is truncated?
- Is the backbone of Figure 2 Qwen2.5-0.5B? In the Alpaca panel, the end points of ZeroPad, HetLoRA, FLoRA and PreLort match the Qwen2.5-0.5B row of Table 2, but the Nested Training curve ends near 2.67, whereas Table 2 gives 3.4447 for Qwen2.5-0.5B.
- Do the FlexLoRA perplexities in Table 2 (e.g., 6.7000, 11.0000) come from the same evaluation code as the other methods?
If the base model scores at or above the fine-tuned models, I would read the MMLU columns as retention rather than adaptation.
-
Can the authors report three seeds for the other Table 1 settings, in particular TinyLlama-1.1B/Dolly (the largest margin) and the ROUGE-L columns? Can they include unrestricted FLoRA and FlexLoRA in the three-seed non-IID comparison (Table 5)? Please also clarify:
- How is FLoRA restricted to a per-round effective rank of 64: by client sampling or by truncation?
- Why do the single-seed 7B results in Table 7 differ so much from the three-seed means in Table 8?
- Why is TinyLlama-1.1B accuracy on 20 Newsgroups higher under the non-IID partition (35.20 ± 0.60, Table 5) than under the IID partition (32.47, Table 1)?
Gains that hold across seeds would raise my score.
-
Can the authors measure cross-client alignment directly, for example the cosine similarity between clients' pseudo-gradients on the shared segment with and without nested training?
- A comparison with FjORD-style training that samples one prefix rank per step would show whether training all prefixes is needed.
- Please report training-only time or FLOPs per method. By Eq. 1, the 10-client setting needs 23 forward and backward passes per batch across clients instead of 10, more than the roughly 40% overhead in Table 9 suggests.
- Please also give the client participation rate per round.
- Please discuss DyLoRA (Valipour et al., 2023), which trains LoRA under sampled rank truncation in the centralized setting.
These answers would change my view of the paper's novelty.
6 comments
Response to Reviewer Nxo7 (2/5)
Weakness 2: The step beyond FjORD and HeteroFL rests on a misalignment argument that the paper does not test: the non-identifiability argument is weak, misalignment is never measured, and the ablations are mixed.
We respectfully keep the identifiability argument but will sharpen it. A LoRA update is invariant to reordering its rank coordinates (more generally, for any invertible ), so standard training does not tie a direction to a rank index or place the most important directions first. Starting each round from the global adapter fixes the initialization, not this ordering; FjORD faces the same situation, since its clients also start from the global model every round, and resolves it with ordered dropout. Table R4 confirms that anchoring alone is not sufficient: with the same initialization and the same segment-wise aggregation, client agreement on the shared segment is 0.300 without nested training vs. 0.633 with it (Qwen2.5-0.5B / 20 Newsgroups). Existing rank-heterogeneous LoRA methods (HetLoRA, FlexLoRA, FLoRA, zero-padding) do not address this ambiguity. To our knowledge, we are the first to identify it in this setting and to resolve it with FjORD-style nestedness: (i) rank index receives supervision from every prefix objective whose width includes (Eq. 2), which orders the coordinates, and (ii) segment is averaged only over clients with . We will rewrite Sections 2 and 3 in these terms.
Alignment measured directly. Figure 3 shows a concentration of importance, which Eq. 2 produces by design; it does not show that client updates agree. We therefore measured the cosine similarity between clients' round updates on the shared segment , with and without nested training, under the same segment-wise aggregation. We use the dense update of the prefix, which does not depend on how each client parameterizes its factors:
Table R4: Mean pairwise cosine similarity between clients' round updates on the shared segment , rounds , seed 42.
| Setting | PreLort | Nested-Agg-Only (no nested training) | Sampled prefix | ZeroPad | FLoRA | FlexLoRA |
|---|---|---|---|---|---|---|
| Qwen2.5-0.5B / 20 Newsgroups (10 clients, ) | 0.633 | 0.300 | 0.718 | 0.166 | 0.075 | 0.250 |
| TinyLlama-1.1B / Dolly (5 clients, ) | 0.513 | 0.159 | 0.540 | 0.014 | 0.002 | 0.343 |
| Qwen3-8B-Base / 20 Newsgroups (10 clients, ) | 0.556 | 0.239 | 0.593 | 0.127 | 0.080 | 0.230 |
Under the same segment-wise aggregation, nested training raises client agreement on the shared segment by about 2-3 (0.300 0.633, 0.159 0.513, 0.239 0.556), well above ZeroPad and FLoRA (); the sampled prefix gives similar agreement, so the alignment comes from nested training itself.
All prefixes vs. one sampled prefix per step: see Table R3 under Question 3.
Ablations. We agree that they are mixed in Table 1: Nested-Agg-Only is the best entry on Qwen2.5-0.5B / 20 Newsgroups, and on Alpaca its prefix perplexities are within about 0.01 of PreLort from rank 4 to rank 32 (Fig. 2). Over three seeds (Tables R1a and R1b), PreLort is above both Nested-Agg-Only and Nested-Training-Only in every setting we reran; the three-seed Qwen2.5-0.5B / 20 Newsgroups comparison is in progress (see Weakness 1). We will describe Nested-Agg-Only as unstable over rounds (Fig. 4) but competitive in some settings, rather than as uniformly inferior.
Response to Reviewer Nxo7 (3/5)
Question 1: What is the MMLU protocol, and how does it relate to the held-out test sets in Appendix B? Please add the base model under the same protocol, and clarify which model is evaluated, the LoRA scaling under truncation, the backbone of Figure 2, and the FlexLoRA perplexities in Table 2.
MMLU protocol. Accuracy is measured on MMLU, as stated in Section 5.1; Appendix B incorrectly says each dataset's held-out test set, and we will correct it. Models are fine-tuned on Dolly or Alpaca and then evaluated zero-shot, without any MMLU training data, on a fixed 1,444-question subset of the MMLU test set covering all 57 subjects (the subset used in FLoRA's evaluation code). The model generates a short answer, which is counted as correct if it contains the gold option, and we report the mean of per-subject accuracies. The Dolly and Alpaca test sets are used only for ROUGE-L and perplexity.
Base model. Under the same protocol and evaluation code:
Table R2: Base models without federated fine-tuning, same evaluation code as Table 1.
| Backbone | MMLU (%) | 20 Newsgroups Acc. (%) | ROUGE-L Dolly | ROUGE-L Alpaca |
|---|---|---|---|---|
| TinyLlama-1.1B (base) | 33.21 | 0.74 | 23.33 | 23.90 |
| Qwen2.5-0.5B (base) | 35.35 | 0.54 | 24.50 | 27.03 |
Fine-tuning improves the task metrics (PreLort: ROUGE-L +4.3 to +9.9; 20 Newsgroups accuracy from <1% to 32.5–39.5%), while MMLU stays within a few points of the base model; we agree that MMLU here measures retention, and will present it as such.
Which model is evaluated. The server's global model after the final round, never the clients' local models. For PreLort, this is the global adapter at the maximum client rank on the frozen backbone; for FLoRA, it is the backbone with the stacked updates merged.
LoRA scaling under truncation. All adapters use , so the scaling factor is the same for every client and for the global adapter. Truncation, both in Figure 2 and in the sub-rank passes of Eq. 2, zeroes the tail rank dimensions in place, so the scaling stays at 2 for every prefix.
Figure 2. The backbone is Qwen2.5-0.5B. The reviewer is right that the Nested Training curve in the Alpaca panel does not match Table 2: its end point (about 2.67) is the TinyLlama-1.1B value (2.6699), so this curve was plotted from the wrong run. We will regenerate the figure from the Qwen2.5-0.5B run.
FlexLoRA perplexities. They come from the same evaluation code as the other methods; the FlexLoRA entries in Table 2 were transcribed with two decimals. We will report the full-precision values.
Response to Reviewer Nxo7 (1/5)
We thank the reviewer for an exceptionally careful reading, for recognizing that the method is simple and honestly scoped, and for highlighting the evidence for low-rank clients in Figure 2 and in the non-IID results. Several of the reviewer's observations identified errors in the paper, which we correct below.
We ran three new seeds for TinyLlama-1.1B / Dolly, the setting with the largest margin in Table 1 (accuracy and ROUGE-L), and for TinyLlama-1.1B / 20 Newsgroups, where FLoRA is best in Table 1. Qwen2.5-0.5B / Dolly already has three seeds in Table 4 (Appendix F.1), repeated below together with its ROUGE-L. All three-seed results, including Table 4, are new runs, separate from the single run reported in Table 1. Three-seed runs for the remaining Table 1 settings (Alpaca on both backbones and Qwen2.5-0.5B / 20 Newsgroups) are in progress. We will post them here if they finish before the end of the discussion period, and will include them in the revised paper in any case.
Table R1a: Accuracy (%), mean ± std over three seeds, same configuration as Table 1. The Qwen2.5 / Dolly column is Table 4.
Table R1b: ROUGE-L, mean ± std over three seeds. Qwen2.5 / Dolly uses the same runs as Table 4.
Over three seeds, PreLort has the highest mean accuracy on TinyLlama-1.1B / Dolly (33.01 ± 0.69 vs. 29.70 ± 1.89 for the best baseline) and the highest ROUGE-L (29.44 ± 0.12 vs. 28.96 ± 0.19 for FLoRA), so this gain holds across seeds. On Qwen2.5-0.5B / Dolly, PreLort also has the highest mean accuracy and ROUGE-L, but both margins (0.31 over FLoRA and 0.15 over FlexLoRA) are within one standard deviation. On TinyLlama-1.1B / 20 Newsgroups, FLoRA remains best (33.85 ± 0.43 vs. 32.67 ± 0.49), as in Table 1. PreLort is above FlexLoRA and above both ablations in every column.
Tables 7 and 8. The reviewer is correct that the single-seed run in Table 7 cannot be one of the three seeds in Table 8. Table 8 pooled runs from two different data partitions, and Appendix F.3 mislabels the setting. We replace both tables with a new experiment on a single partition, with three seeds and 10 communication rounds, on a recent backbone:
Table R5: Qwen3-8B-Base on 20 Newsgroups, 10 clients, ranks [64,32,16,16,8,8,4,4,4,4], 10 communication rounds × 1 local epoch, mean ± std over three seeds.
On Qwen3-8B-Base with three seeds and 10 rounds, PreLort obtains the highest accuracy (64.23 ± 0.56%), 0.67 points above FlexLoRA and well above FLoRA (+11.3), HetLoRA (+24.5) and ZeroPad (+34.2).
Metric. Section 5.1 is correct: Dolly and Alpaca accuracy is MMLU accuracy after fine-tuning, and Appendix B is wrong. The MMLU protocol and the base-model results are given under Question 1. We will remove "consistently outperforms" from the abstract and report the per-setting findings above.