acceptodds
Under review as a conference paper at ICLR 2027

When Is Low-Rank Pretraining Actually Efficient? A Resource-Aware Benchmark

Abstract

Low-rank methods offer a broadly applicable way to improve the efficiency of LLM pretraining by reducing model parameters, optimizer memory, or FLOPs. Yet their practical efficiency remains difficult to compare. Existing studies use different hardware, model scales, implementations, and training budgets, and often report intermediate savings without showing how they translate into end-to-end memory and training time. Therefore, a meaningful comparison requires a shared training setup, explicit control of the training budget, and direct measurement of both model quality and realized system costs. To provide such a comparison, we introduce the Low-rank Efficient Pretraining Benchmark (LR-EPB), which evaluates ten low-rank pretraining methods that place low-rank structure in model weights, optimizer states, or weight updates, on LLaMA-style models from 60M to 1B parameters. Across 130 runs and approximately 13,000 equivalent A100-40 GPU-hours, we compare model quality under matched-token and matched-FLOPs budgets and measure training FLOPs, peak memory, throughput, and serving latency. Our results show that low-rank efficiency is better characterized as a resource profile than by a single metric. Under matched FLOPs, low-rank weight methods can trade lower per-token cost for more training data and close much of their quality gap to full-rank training. Memory savings depend on which components are reduced, while FLOP reductions translate into wall-clock speed more effectively as model scale grows. Optimizer-state methods show a different profile, with modest memory savings and additional execution overhead despite nearly unchanged FLOPs. During serving, direct execution and V/O absorption favor different workloads. V/O absorption reuses the trained low-rank factors to reduce decoding latency and memory at large batch sizes without retraining. LR-EPB provides resource-specific guidance for selecting low-rank pretraining methods under different training and serving constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter Efficient Pretraining, Memory Efficient Pretraining, Efficient Pretraining, Parameter and Memory Efficient Pretraining

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_A567OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a benchmark, coined LR-EPB, comparing ten low-rank pretraining methods and two dense references on LLaMA-style models from 60M to 1B parameters.

It evaluates model quality under matched-token and matched-FLOPs budgets and measures training memory, throughput, and serving latency.

The results show that lower FLOPs allow more training tokens and can narrow the quality gap, while actual memory savings and training speed depend on the method, model scale, and workload.

The paper also evaluates V/O absorption, which reuses trained low-rank factors to reduce value-cache storage and improve decoding speed at large batch sizes without retraining.

Critical strengths

1, The benchmark makes the effect of the training budget clear. Comparing the same methods under matched-token and matched-FLOPs budgets shows how additional training tokens can narrow the quality gap of low-rank models. This helps explain why evaluations using different budgets can reach different conclusions.

2, The paper measures whether theoretical savings translate into practical benefits. It reports peak memory and training throughput alongside FLOPs, showing that optimizer-state compression can provide limited memory savings when activations dominate, and that fewer FLOPs do not necessarily mean faster training.

3, The serving analysis provides a useful way to reuse the trained low-rank structure. V/O absorption preserves the model computation without retraining and reduces value-cache storage. The paper also identifies its workload dependence: the additional arithmetic can hurt smaller batches, while reduced cache traffic benefits larger batches.

Critical weaknesses

1, Each configuration is trained with one seed, and the methods use different tuning procedures and learning-rate schedules. The tolerance estimated from matched-token and matched-FLOPs runs does not establish variation across seeds, so small differences are insufficient to identify a clear winner. Repeating the key comparisons with comparable tuning budgets would make the method recommendations more convincing.

2, For the largest model at batch 128, a single BF16 KV cache should occupy about 15 GiB, while the reported direct-execution peak is 46.2 GiB. Temporary allocations may explain this difference, but their contribution is unclear. Similarly, the 2.1–2.2× decoding speedup cannot be explained by the 1.6× KV-cache reduction alone. A memory breakdown and kernel profile would help establish how much of the gain comes from absorption and how much depends on the implementation.

Questions for authors

1, For the largest model at batch 128 and context length 640, one BF16 KV cache should occupy about 15 GiB. Including the factorized weights gives approximately 16.1 GiB, compared with the reported peak of 46.2 GiB. Doubling the batch from 64 to 128 increases memory by 22.5 GiB, three times the expected cache increase. Could the authors specify the cache dtype and separate persistent storage from temporary allocations or retained copies?

2, V/O absorption reduces total KV-cache storage by a factor of 1.6, while the reported decoding speedup reaches 2.1–2.2× and value-aggregation arithmetic increases. These results are possible, but cache compression alone does not explain them. Could the authors provide kernel timings and memory-traffic measurements to identify the additional sources of improvement?

3, The reported optimizer-state memory of approximately 5 GiB for 1.339B parameters matches two BF16 moment tensors; two FP32 moment tensors would require approximately 10 GiB. Could the authors specify parameter, gradient, optimizer-state, and master-weight precision for each method? This detail affects the interpretation of the memory-saving results.

4, The main text describes FLOP measurement throughout training, while Appendix B.1 describes profiling early updates and extrapolating. Could the authors clarify how this accounts for projection refreshes, Riemannian steps, ReLoRA merges, and its initial dense-training phase? Reporting ordinary-step costs, periodic-operation costs, and their frequencies would make the accounting easier to verify.

5, For 350M LORO at batch 64, direct decoding takes 28.0 ms in Table 15 and 23.6 ms in Table 16, a difference of approximately 16%. The paper acknowledges separate profiling sessions, but this variation could affect comparisons near the latency crossover. Could the authors confirm identical execution settings and report variation across repeated measurements?

8 comments
AuthorsOpenReview ↗

Response to Reviewer A567: W1, W2-part1 and Q1

Thank you for your thorough review and detailed technical comments. Your questions have been particularly helpful in refining our experimental analysis and measurements. We address each point below with additional results and clarifications. Please let us know if you have any further concerns.

W1. Each configuration is trained with one seed, and the methods use different tuning procedures and learning-rate schedules. The tolerance estimated from matched-token and matched-FLOPs runs does not establish variation across seeds, so small differences are insufficient to identify a clear winner. Repeating the key comparisons with comparable tuning budgets would make the method recommendations more convincing.

We revise the method recommendations to avoid selecting a unique winner based on small PPL differences. We also recheck selected configurations with noticeably different PPL despite similar training budgets. The revised Full-size reference improves from 17.43 to 16.97 at 130M and from 13.40 to 13.15 at 350M. The updated comparisons are shown below.

Scale / BudgetFull-sizeSinkGDFiraAPOLLOAlice
130M / Matched tokens16.9717.1416.9817.5717.53
130M / Matched FLOPs16.9716.8016.6417.1617.41
350M / Matched tokens13.1512.5412.7212.9612.98
350M / Matched FLOPs13.1512.6212.9813.0313.11

At 350M, Fira, APOLLO, and Alice differ by at most 0.26 PPL under matched tokens and 0.13 under matched FLOPs. At 130M under matched tokens, Fira (16.98) is also nearly identical to Full-size (16.97). These differences do not support unique winner rankings.

Regarding tuning, Full-size uses a six-value learning-rate search at each model scale, while the other methods follow their authors' recommended configurations, with adjustments for training stability where needed. We report these settings in App. A.3–A.4. The 0.5-PPL tolerance is based on near-equivalent runs under the two budgets, rather than seed-to-seed variation. Therefore, we will update Tab. 1 to group methods with comparable quality and provide recommendations based on their measured memory and throughput trade-offs, rather than small differences in PPL.

W2-part1 and Q1. W2-part1: For the largest model at batch 128, a single BF16 KV cache should occupy about 15 GiB, while the reported direct-execution peak is 46.2 GiB. Temporary allocations may explain this difference, but their contribution is unclear. Q1: For the largest model at batch 128 and context length 640, one BF16 KV cache should occupy about 15 GiB. Including the factorized weights gives approximately 16.1 GiB, compared with the reported peak of 46.2 GiB. Doubling the batch from 64 to 128 increases memory by 22.5 GiB, three times the expected cache increase. Could the authors specify the cache dtype and separate persistent storage from temporary allocations or retained copies?

We identify the source of the inflated memory usage and conduct additional serving measurements. In our original implementation, the KV cache grows through torch.cat, which temporarily allocates new tensors while retaining the previous cache. Moreover, the previous request's final cache remains in memory until the current request finishes. Consequently, multiple cache-sized tensors can coexist, leading to an inflated peak. In a controlled experiment at batch 128, retaining the previous request's cache increases peak memory from 33.04 to 48.04 GiB, an additional 15.00 GiB.

We correct this by preallocating BF16 K/V buffers and updating them in place. Each request reuses the buffers, and attention accesses only the valid cache prefix. We repeat all 53 serving configurations on the same NVIDIA A800 using the corrected cache implementation. The following table reports the results for 1B CoLA. All values are in GiB.

BatchOriginal direct peakCorrected direct peakCorrected absorbed peakStatic KV storage (direct / absorbed)
6423.6711.799.487.49 / 4.68
12846.1722.4117.4114.98 / 9.36

The corrected measurements substantially reduce the reported peak memory. At batch 128, direct execution now uses 22.41 GiB instead of 46.17 GiB. V/O absorption further reduces the peak to 17.41 GiB, while the KV-cache storage decreases from 14.98 to 9.36 GiB. The remaining memory beyond the KV cache includes model weights and temporary allocations.

We will update the serving memory results and clarify the cache allocation and measurement protocol in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer A567: W2-part2, Q2 and Q3

W2-part2 and Q2. W2-part2: The 2.1–2.2× decoding speedup cannot be explained by the 1.6× KV-cache reduction alone. A memory breakdown and kernel profile would help establish how much of the gain comes from absorption and how much depends on the implementation. Q2: V/O absorption reduces total KV-cache storage by a factor of 1.6, while the reported decoding speedup reaches 2.1–2.2× and value-aggregation arithmetic increases. These results are possible, but cache compression alone does not explain them. Could the authors provide kernel timings and memory-traffic measurements to identify the additional sources of improvement?

We conduct additional kernel profiling and repeat the serving experiments to identify the sources of the decoding speedup. The results show that the original 2.1–2.2× speedup is partly inflated by two implementation factors.

First, the original direct execution uses default math SDPA, which converts the full BF16 K/V caches to FP32 during decoding. In contrast, V/O absorption uses BF16 matrix multiplications and FP32 softmax, avoiding these conversions. At 1B, batch 128, and context length 640, the 48 conversion kernels account for 41.36 ms per decoding step, corresponding to approximately 15 GiB of logical reads and 30 GiB of logical writes based on tensor sizes. With preallocated KV caches, enabling BF16 math reduction eliminates these conversions and reduces direct decoding latency from 92.82 to 30.98 ms per step. Absorbed execution takes 28.08 ms in the same single-step profile.

Second, dynamic KV-cache allocation also contributes to the original speedup. At 1B and batch 128, cache appending takes 29.97 ms for direct execution and 17.83 ms for absorption. With preallocated KV buffers, this overhead decreases to approximately 0.2 ms for both paths.

We therefore repeat all 53 serving configurations using preallocated BF16 KV caches and improved BF16 math settings. The following table reports the corrected time per output token (TPOT) in milliseconds.

Scale / BatchModelFull-sizeDirectAbsorbedSpeedup vs. DirectSpeedup vs. Full-size
350M / 128CoLA20.3726.4025.891.02×0.79×
350M / 128LORO20.3731.2630.061.04×0.68×
350M / 256CoLA33.8535.0624.641.42×1.37×
350M / 256LORO33.8535.6928.901.24×1.17×
1B / 128CoLA38.8537.3625.831.45×1.50×
1B / 128LORO38.8537.9830.261.26×1.28×

The corrected results confirm that V/O absorption still provides substantial decoding speedups at larger batch sizes. At 1B and batch 128, CoLA and LORO achieve 1.50× and 1.28× the decoding speed of Full-size, respectively. At 350M, absorption becomes beneficial at batch 256, while it remains slower than Full-size at batch 128. These results support our finding that the benefit of V/O absorption depends on the trade-off between reduced KV-cache traffic and additional attention-value aggregation. We will update the serving results and clarify the execution settings in the revised paper.

Q3. The reported optimizer-state memory of approximately 5 GiB for 1.339B parameters matches two BF16 moment tensors; two FP32 moment tensors would require approximately 10 GiB. Could the authors specify parameter, gradient, optimizer-state, and master-weight precision for each method? This detail affects the interpretation of the memory-saving results.

Our implementation uses BF16 for model parameters, gradients, and persistent optimizer states across all twelve methods. SinkGD maintains no optimizer states, and no separate FP32 master-weight copies are maintained.

For the 1B Full-size model, the reported 5.0 GiB optimizer-state memory consists of AdamW's first- and second-moment tensors. Each tensor contains 1.339B BF16 elements, giving a total of 2×1.339×109×2/230≈4.992 \times 1.339 \times 10^9 \times 2 / 2^{30} \approx 4.99 GiB. Thus, the reported memory savings are measured under a consistent BF16 training configuration. We will explicitly document these precision settings in App. B.3.

AuthorsOpenReview ↗

Response to Reviewer A567: Q4 and Q5

Q4. The main text describes FLOP measurement throughout training, while Appendix B.1 describes profiling early updates and extrapolating. Could the authors clarify how this accounts for projection refreshes, Riemannian steps, ReLoRA merges, and its initial dense-training phase? Reporting ordinary-step costs, periodic-operation costs, and their frequencies would make the accounting easier to verify.

Our FLOP accounting includes both regular training updates and method-specific periodic operations. Specifically, we use PyTorch's FlopCounterMode to profile the first 500 optimizer updates, including forward and backward passes, optimizer steps, and periodic operations occurring within this window. We also register additional FLOP-counting formulas for SVD, QR factorization, and linear solves. For operations not captured by the counter, we include explicit FLOP corrections. The average FLOPs per update, including amortized periodic costs, is then used to track the cumulative training budget.

The periodic operations and their frequencies are summarized below.

MethodPeriodic operationFrequency
GaLore, GoLore, Fira, Alice, APOLLOProjection or basis refreshEvery 200 updates
LOROExact Riemannian stepEvery 500 updates
ReLoRAAdapter mergingEvery 2,000 updates

ReLoRA is handled separately because it contains different training phases. We calculate FLOPs for its initial dense-training phase and subsequent low-rank adapter phase separately, and include the cost of periodic adapter merges. The initial dense-training phase lasts 2,000, 4,000, and 10,000 updates at 60M, 130M, and 350M, respectively.

Thus, the throughout training description in Sec. 3 refers to tracking cumulative FLOPs throughout training, rather than continuously running the FLOP counter. We will clarify this distinction in Sec. 3 and App. B.1 and provide the ordinary-step and periodic-operation FLOP costs alongside their frequencies.

Q5. For 350M LORO at batch 64, direct decoding takes 28.0 ms in Table 15 and 23.6 ms in Table 16, a difference of approximately 16%. The paper acknowledges separate profiling sessions, but this variation could affect comparisons near the latency crossover. Could the authors confirm identical execution settings and report variation across repeated measurements?

We conduct a unified remeasurement of all 53 serving configurations to ensure consistent comparisons. The original direct-execution values of 28.0 and 23.6 ms are measured on different machines at different times. In the new experiments, we use the same physical NVIDIA A800, preallocated BF16 KV caches, and a consistent measurement protocol. Paired direct and absorbed runs also use identical model weights and fixed input token sequences.

For 350M LORO at batch 64, with a 512-token prompt and 128 generated tokens, we obtain the following results:

MeasurementDirect TPOT (ms)Absorbed TPOT (ms)
Original Tab. 1528.0–
Original Tab. 1623.626.5
Unified remeasurement31.62 ± 0.5528.78 ± 0.92

The updated values report mean ± standard deviation across 20 measured requests.

Under the unified protocol, V/O absorption achieves a 1.10× speedup over direct execution at this configuration. We will update Fig. 4 and Tabs. 15–16 using the same measurement campaign and revise the latency crossover analysis accordingly.

Reviewer_fPyaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The work studies the condition that makes low-rank pre-training efficient and provides four insights. Specifically, the paper proposes matched-token and matched-FLOPs budgets, with perplexity, memory, and throughput as measures. The key findings include (1) When fixing the FLOP budget, low-rank weight approaches can reduce the perplexity, gradient and optimizer states are more important in this setting; (2) The benefit of low-rank in terms of memory consumption changes with the components' memory usage; (3) FLOPs' scalability varies for different low-rank components; and (4) Low-rank approaches need to be tailored for different execution paths, i.e., prefill and decode in serving and low-rank-aware training.

Critical strengths
  1. The paper provided comprehensive measures of the practical system performance. For example, peak memory usage by component and throughput are reported.

  2. Matched-token and -FLOPs are well motivated.

  3. Table 1 presents clear and useful guidance to understand the logic of this benchmark design.

Critical weaknesses
  1. The system used in the experiment has narrow representations. Only single-GPU profiling with a sequence length of 256 was used in the experiments, which is not enough for a benchmark workload.

  2. Limited novelty. Some findings have been reported in existing work. For example, the V/O absorption is a direct application of MLA. Thus, the claim that V/O absorption reduces the value-cache traffic in the finding is weak.

Questions for authors
  1. Are memory-efficient approaches such as FlashAttention used when measuring different components' memory usage?

  2. How do low-rank approaches perform with larger models, such as 8B, 30B, 70B, using multiple GPUs? It would be more interesting for a benchmark study that observes performance at a larger scale.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fPya: W1 and W2

Thank you for taking the time to review our paper and for your constructive comments on the benchmark's scope and contributions. We address your concerns point by point below and provide additional experimental results where relevant. Please kindly let us know whether you have any further concerns.

W1. The system used in the experiment has narrow representations. Only single-GPU profiling with a sequence length of 256 was used in the experiments, which is not enough for a benchmark workload.

There appears to be a misunderstanding regarding the sequence length used in our experiments. Our paper already evaluates system performance across multiple sequence lengths, not only 256. Specifically, Appendix B.4 and Tab. 11 report training throughput at sequence lengths 256, 512, 1024, and 2048, with the number of tokens per profiled update held constant. In addition, Appendix D.4 and Fig. 7 examine how sequence length and batch size affect memory consumption.

These experiments reveal meaningful workload-dependent differences. For example, at 1B, CoLA achieves 1.70× the throughput of Full-size at sequence length 256, which decreases to 1.43× at length 2048. LORO follows a similar trend, decreasing from 1.66× to 1.35×. Thus, our benchmark not only covers different sequence lengths but also shows how they affect the practical efficiency of low-rank methods.

Regarding hardware, our pretraining experiments use four A100 GPUs, while system profiling related experiments is performed on a single GPU. The single-GPU setting allows us to compare the execution overhead, throughput, and memory consumption of different methods under the same hardware conditions, without confounding effects from distributed communication.

Therefore, our paper already covers multiple workload configurations and provides controlled measurements of the resource trade-offs among low-rank methods.

W2. Limited novelty. Some findings have been reported in existing work. For example, the V/O absorption is a direct application of MLA. Thus, the claim that V/O absorption reduces the value-cache traffic in the finding is weak.

V/O absorption is not presented as a new algorithm in our paper. Sec. 6 explicitly states that we follow the weight-absorption idea from MLA. However, the setting and motivation are different. MLA is designed with joint K/V compression and decoupled RoPE to reduce inference costs. In contrast, methods such as CoLA and LORO learn low-rank weight factors during pretraining. We show that these already-trained factors can be reused to compress the value cache during serving by reorganizing only the V/O projections, without retraining or modifying the original attention architecture. Thus, the low-rank structure learned for training efficiency can provide an additional benefit during deployment.

Importantly, this serving benefit is not guaranteed by V/O absorption itself. Although absorption reduces value-cache storage, it also increases the computation required for attention-value aggregation, as analyzed in App. E.2. Therefore, whether it improves decoding speed depends on the workload. Our updated serving experiments demonstrate this trade-off:

Model / BatchCoLA speedupLORO speedup
350M / 1280.79×0.68×
350M / 2561.37×1.17×
1B / 1281.50×1.28×

Decoding speed relative to Full-size with V/O absorption. Values below 1 indicate slower decoding.

These results show that V/O absorption can be slower than Full-size at moderate batch sizes but substantially faster at larger batch sizes. Finding 4 characterizes this workload-dependent behavior and its implications for deploying low-rank pretrained models, rather than claiming novelty in the absorption operation itself.

More broadly, V/O absorption is one part of LR-EPB. Our main contribution is a unified, resource-aware characterization of low-rank pretraining, which compares ten low-rank methods under matched-token and matched-FLOPs budgets and measures their training memory, throughput, and serving efficiency. This joint evaluation reveals how improvements in one resource may not translate into improvements in another, and how the practical benefits of low-rank methods depend on the workload. Together, these findings provide resource-specific insights and guidance that go beyond any individual efficiency metric or serving technique.

AuthorsOpenReview ↗

Response to Reviewer fPya: Q1 and Q2

Q1. Are memory-efficient approaches such as FlashAttention used when measuring different components' memory usage?

The submitted memory measurements use math SDPA. To examine whether our findings hold with more efficient attention implementations, we conducted additional experiments using Flash SDPA. We compared math SDPA and Flash SDPA for Full-size, SinkGD, GaLore, Fira, CoLA, and LORO at sequence lengths 512 and 1024. All experiments were conducted on the same NVIDIA A800 GPU, with identical microbatch and gradient accumulation settings across methods within each workload. We also verified the dispatched attention kernels.

The following table reports the results at 350M. Each entry shows throughput (thousand tokens/s) / peak allocated memory (GiB).

Method512 Math512 Flash1024 Math1024 Flash
Full-size22.90 / 20.7347.88 / 13.6015.46 / 26.7346.76 / 13.60
SinkGD22.29 / 19.5845.74 / 12.4615.35 / 25.5945.67 / 12.46
GaLore22.24 / 19.9545.12 / 12.8315.30 / 25.9545.47 / 12.83
Fira21.91 / 20.0943.76 / 12.9815.23 / 26.0944.70 / 12.98
CoLA24.81 / 19.6657.03 / 12.5516.28 / 25.6655.28 / 12.55
LORO23.42 / 20.0150.34 / 12.8915.76 / 26.0149.74 / 12.90

Flash SDPA substantially improves both memory efficiency and throughput. For example, at sequence length 1024, Full-size peak allocated memory decreases from 26.73 to 13.60 GiB, while throughput increases from 15.46K to 46.76K tokens/s. Similar improvements are observed across the other methods.

Importantly, the relative efficiency differences between methods remain under Flash SDPA. At sequence length 1024, CoLA and LORO achieve 18.2% and 6.4% higher throughput than Full-size, respectively, while GaLore and Fira are 2.8% and 4.4% slower. In terms of memory, CoLA and LORO use 7.7% and 5.1% less peak allocated memory than Full-size. These results show that efficient attention implementations substantially change the absolute resource costs, while low-rank methods retain distinct efficiency profiles. They further support our finding that practical efficiency depends on both the low-rank method and the execution workload.

Q2. How do low-rank approaches perform with larger models, such as 8B, 30B, 70B, using multiple GPUs? It would be more interesting for a benchmark study that observes performance at a larger scale.

Evaluating 8B–70B models requires substantial computational resources and is not feasible within the rebuttal period.

More importantly, LR-EPB already provides a systematic and meaningful comparison within its evaluated scope. LR-EPB includes 130 training runs, approximately 13,000 equivalent A100 GPU-hours, across four model scales from 60M to 1B, comparing ten low-rank methods and two reference methods under matched-token and matched-FLOPs budgets. This scale range is consistent with that of the related benchmark by Glentis et al. (2025). Moreover, our experiments already reveal substantial scale-dependent differences in training quality and resource efficiency. Therefore, extending the benchmark to larger distributed models is a worthwhile direction, but is not necessary to establish the findings reported in our paper.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer fPya,

Thank you for your questions about workload coverage and V/O absorption. As the discussion period draws to a close, we wanted to follow up on the math and Flash SDPA comparisons at longer sequence lengths and our clarification of when V/O absorption helps decoding. We would appreciate your thoughts on these updates and would be happy to discuss any remaining questions.

Reviewer_DBJBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents LR-EPB, a benchmark of low-rank LLM pretraining. Ten low-rank methods (four that factorize weights, one that trains low-rank updates, and five that compress optimizer states) are compared with full-rank AdamW and the stateless SinkGD on LLaMA-style models from 60M to 350M parameters trained on FineWeb-Edu, with a representative subset at 1B, under matched-token and matched-FLOPs budgets with profiled FLOPs. Peak memory, throughput and serving latency are measured on one A800 GPU, including a V/O absorption path for factorized checkpoints. The main findings: the budget changes the quality comparison for methods that cut FLOPs per token; optimizer-state compression saves little peak memory when activations dominate; FLOP savings turn into training speed mainly at 1B; and V/O absorption speeds up large-batch decoding. Table 1 turns these into per-resource recommendations.

Critical strengths
  1. The budget is explicit and profiled. All methods share data, training and evaluation code, and hardware; each is trained under both budgets; and FLOPs are profiled directly, including SVD, QR and solve calls (App. B.1). My architecture-level arithmetic agrees with the reported dominant costs: from Table 3 I get about 2.09 GFLOPs per token for Full-size at 350M and 0.99 for rank 256, matching Fig. 2 and the 2.11x token multiplier in Table 2. The LORO example (a 27% gap at matched tokens versus 2% at matched FLOPs, p. 2) shows how a single budget can mislead.

  2. It ties structural savings to measured end-to-end cost. At 1B the separately measured activation peak (26.2 GiB) is large relative to the 38.0 GiB reserved peak (Fig. 3a), which explains why removing optimizer states lowers peak memory by only about 15%. V/O absorption lets factorized checkpoints decode 1.5 to 2.2x faster than Full-size at batch 64 to 128 at 1B, with 30% to 36% less peak generation memory than direct execution of the same factorized model and no retraining (Table 16). These are practical results for anyone choosing a method.

Critical weaknesses
  1. Table 1's single-winner recommendations are finer than the paper's own tie rule allows. The paper treats perplexity gaps below 0.5 at 130M and 350M as ties, based on paired runs (p. 5, App. C.2), yet the three best low-rank methods at 350M are within 0.26 of each other at matched tokens and within 0.13 at matched FLOPs, and at 130M the best low-rank run is 0.45 below Full-size at matched tokens (Table 2). Table 1 still names a single winner, and all runs use one seed, so these orderings are not established; the tolerance itself comes from paired runs under different budgets with a fixed seed, not from seed variation. At 1B, comparisons with the dense reference rest on an unusual configuration: Full-size improves little from 350M to 1B (13.40 to 12.49, while SinkGD goes from 12.54 to 10.08), and it alone uses cosine decay with warm restarts (cycle 500, or 200 at 1B, Table 4). This affects improvements relative to AdamW, not the ordering among low-rank methods or the memory and latency ratios. Marking ties in Table 1, a few seeds for the leading methods, and a check of the 1B Full-size schedule would address this.

  2. The system measurements are useful at their stated workload, but the practical recommendations need closer alignment with the training setup. Throughput (Table 11) is measured with one optimizer update per 8,192 tokens on one GPU (App. B.4), while the quality runs use 32,768 tokens per GPU per update with accumulation (App. A.2), so per-update costs (projection, refresh, the optimizer step) weigh about 4x more in profiling than in training; how much of the 6% to 24% slowdown of the optimizer-state methods remains under the training accumulation is not measured. Likewise, activation checkpointing is applied only to the factorized models. The paper reports both checkpointed and standard factorized rows and attributes the extra saving to checkpointing, but Table 1's device-memory advice lacks the memory and throughput trade-off of Full-size and optimizer-state methods with standard or selective checkpointing. Targeted re-measurement of both would make the guidance reliable.

Questions for authors

quality runs (32,768 tokens per GPU per update), at least for Full-size, SinkGD and the optimizer-state methods at 350M and 1B, with forward/backward, optimizer-step and refresh time reported separately? How would the results change Table 1's wall-clock guidance?

  1. Could you add Full-size and one optimizer-state method with standard or selective activation checkpointing to Tables 12 and 13? Please also state the optimizer-state dtypes and which tensors the 5.0 GiB at 1B includes; it is consistent with two 2-byte moments for 1.34B parameters.

  2. For the 1B Full-size reference, could you give the validation results across the six-value learning-rate grid, the selection criterion and the actual learning-rate trace? How do the short restart cycles interact with the initial warmup and the budget-fraction schedule, and how sensitive is the 1B result to a standard cosine schedule?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DBJB: W1 and Q3

We appreciate your careful review and detailed suggestions regarding the experimental protocol and resource measurements. Your comments have helped us further strengthen our evaluation and comparisons. We provide point-by-point responses and additional experimental results below. Please let us know if you have any further concerns.

W1-part1. Table 1's single-winner recommendations are finer than the paper's own tie rule allows. The paper treats perplexity gaps below 0.5 at 130M and 350M as ties based on paired runs, yet the three best low-rank methods at 350M are within 0.26 of each other at matched tokens and within 0.13 at matched FLOPs. Table 1 still names a single winner, and all runs use one seed, so these orderings are not established.

We will revise Tab. 1 to report methods with comparable performance rather than selecting a single winner based on small PPL differences. We also rechecked selected configurations where runs with similar token budgets showed noticeable PPL differences. The revised Full-size reference improves from 17.43 to 16.97 at 130M and from 13.40 to 13.15 at 350M. The following table summarizes the updated comparisons.

Scale / BudgetFull-sizeSinkGDFiraAPOLLOAlice
130M / Matched tokens16.9717.1416.9817.5717.53
130M / Matched FLOPs16.9716.8016.6417.1617.41
350M / Matched tokens13.1512.5412.7212.9612.98
350M / Matched FLOPs13.1512.6212.9813.0313.11

At 350M, Fira, APOLLO, and Alice differ by at most 0.26 PPL under matched tokens and 0.13 under matched FLOPs. At 130M under matched tokens, Fira (16.98) is also nearly identical to the revised Full-size reference (16.97). These small differences do not justify unique winner rankings.

The 0.5-PPL threshold in App. C.2 is an empirical comparison tolerance based on near-equivalent runs under the two budgets. We will clarify this and group methods with comparable PPL. We will also distinguish low-rank methods from the Full-size and SinkGD references in Tab. 1.

W1-part2 and Q3. W1-part2: At 1B, comparisons with the dense reference rest on an unusual configuration: Full-size improves little from 350M to 1B (13.40 to 12.49) ... and it alone uses cosine decay with warm restarts. Q3: For the 1B Full-size reference, could you give the validation results across the six-value learning-rate grid, the selection criterion and the actual learning-rate trace? How do the short restart cycles interact with the initial warmup and the budget-fraction schedule, and how sensitive is the 1B result to a standard cosine schedule?

The reported 1B Full-size result is the best among the configurations we have evaluated. We search six learning rates, {10−4,3×10−4,10−3,3×10−3,10−2,3×10−2}\{10^{-4}, 3\times10^{-4}, 10^{-3}, 3\times10^{-3}, 10^{-2}, 3\times10^{-2}\}, and select the configuration with the lowest final validation PPL. The best result is 12.49 at a learning rate of 3×10−43\times10^{-4}, while learning rates above 10−310^{-3} lead to divergence. Our tuning protocol does not favor low-rank methods. Full-size receives an explicit six-value learning-rate search, while low-rank methods primarily follow their authors' recommended configurations. We also explore additional hyperparameter configurations for Full-size, but none of the completed runs improves this result.

Our training schedule uses an initial warmup covering 10% of the training budget, together with cosine warm restarts using a cycle length of 200 updates and 5 restart warmup steps. We will clarify the exact learning-rate schedule and report the learning-rate search results.

The Full-size baseline affects the magnitude of the quality gaps relative to AdamW. However, it does not affect the comparisons among low-rank methods or the measured FLOPs, memory, throughput, and serving costs. We are continuing to examine alternative configurations, but full 1B pretraining is computationally expensive, and additional runs may not finish before the rebuttal deadline. We will report any improved results if available.

AuthorsOpenReview ↗

Response to Reviewer DBJB: W2-part1 and Q1

W2-part1 and Q1. W2-part1: Throughput (Table 11) is measured with one optimizer update per 8,192 tokens on one GPU, while the quality runs use 32,768 tokens per GPU per update with accumulation. Thus, per-update costs weigh about 4× more in profiling than in training. How much of the 6% to 24% slowdown of the optimizer-state methods remains under the training accumulation? Q1: Could you repeat the throughput measurements under the training accumulation setting at 350M and 1B, with forward/backward, optimizer-step, and refresh time reported separately? How would the results change Tab. 1's wall-clock guidance?

We conduct additional throughput profiling experiments using the actual training gradient accumulation (GA) settings, with 32,768 tokens per GPU per optimizer update. We also measure forward, backward, optimizer-step, and periodic refresh costs separately. All measurements use a single NVIDIA A800 at sequence length 256 with math SDPA. At 350M and 1B, all methods use microbatch 32 and GA=4.

The following table reports throughput in thousand tokens/s and timing breakdowns in ms per optimizer update. The timing columns show forward / backward / optimizer / refresh, with periodic refresh costs amortized over their configured intervals. “–” indicates that no separate periodic refresh operation is required.

Method350M Throughput350M Timing (ms)1B Throughput1B Timing (ms)
Full-size29.97421 / 661 / 11 / –8.311500 / 2413 / 31 / –
SinkGD28.23421 / 662 / 75 / –8.161499 / 2409 / 107 / –
GaLore27.76421 / 662 / 14 / 837.661499 / 2410 / 58 / 312
Fira26.80421 / 662 / 61 / 787.611499 / 2409 / 85 / 309
CoLA33.34382 / 595 / 6 / –13.67898 / 1478 / 20 / –
LORO30.50419 / 614 / 22 / 1812.261016 / 1492 / 26 / 137

The slowdown of optimizer-state methods is substantially smaller under the training accumulation settings. Across the five optimizer-state methods, throughput is 3.7–10.6% lower than Full-size at 350M and 0.7–8.4% lower than Full-size at 1B. For example, at 350M, GaLore's slowdown decreases from 24.2% under GA=1 to 7.4% under training GA. This is because gradient accumulation spreads the optimizer-step and projection-refresh costs over more tokens.

The throughput profiling in our paper also includes a GPU-to-CPU gradient-norm diagnostic. In the additional experiments, we exclude this diagnostic from both GA=1 and training-GA measurements to eliminate its overhead.

These results lead to updated wall-clock recommendations in Tab. 1. CoLA achieves the highest throughput at both 350M and 1B, reaching 1.11× and 1.65× that of Full-size, respectively, while LORO reaches 1.02× and 1.47×. Meanwhile, optimizer-state methods incur much smaller throughput penalties under the actual training settings. We will report the complete measurements for all methods in the revised appendix.

AuthorsOpenReview ↗

Response to Reviewer DBJB: W2-part2 and Q2

W2-part2 and Q2-part1. W2-part2: Activation checkpointing is applied only to the factorized models. The paper reports both checkpointed and standard factorized rows and attributes the extra saving to checkpointing, but Tab. 1's device-memory advice lacks the memory and throughput trade-off of Full-size and optimizer-state methods with standard or selective checkpointing. Q2-part1: Could you add Full-size and one optimizer-state method with standard or selective activation checkpointing to Tabs. 12 and 13?

We conduct additional experiments comparing standard gradient checkpointing (GCP) across Full-size, SinkGD, optimizer-state methods, and low-rank weight methods. We also update the structure-aware activation checkpointing (AC) implementation to include the MLP and attention output projections.

The following table reports the 1B results on a single NVIDIA A800 with math SDPA, sequence length 256, microbatch 32, and gradient accumulation (GA)=4. Each cell shows peak reserved memory (GiB) / throughput (thousand tokens/s). All methods process 32,768 tokens per GPU per optimizer update.

MethodNo checkpointingStandard GCPUpdated structure-aware AC
Full-size39.59 / 8.3114.02 / 6.05–
SinkGD34.50 / 8.179.26 / 5.98–
GaLore38.00 / 7.6612.15 / 5.70–
Fira36.95 / 7.6111.22 / 5.67–
Vanilla LR34.93 / 12.968.01 / 9.339.81 / 9.66
CoLA34.21 / 13.698.14 / 10.039.97 / 10.04
LORO34.94 / 12.288.02 / 8.9610.06 / 9.28

“–” indicates that structure-aware AC is not applied to the method.

Low-rank weight methods retain substantial memory and throughput advantages under the same standard GCP policy. At 1B, CoLA and LORO use 8.14 and 8.02 GiB, respectively, compared with 14.02 GiB for Full-size, reducing peak memory by approximately 42–43%. Meanwhile, they achieve 1.66× and 1.48× the throughput of Full-size, respectively.

The additional experiments also reveal the memory–throughput trade-off of checkpointing. Both GCP and structure-aware AC substantially reduce peak memory at the cost of additional recomputation. Standard GCP further reduces peak memory by 18–20% compared with structure-aware AC, while maintaining similar throughput. For example, CoLA uses 8.14 GiB with standard GCP versus 9.97 GiB with structure-aware AC, with nearly identical throughput of 10.03K and 10.04K tokens/s.

These results strengthen our device-memory recommendations in Tab. 1. Low-rank weight methods provide substantial additional memory savings even when Full-size and optimizer-state methods use standard checkpointing. We will update Tab. 1 accordingly and include the checkpointing comparisons across all four model scales in the revised appendix.

Q2-part2. Please also state the optimizer-state dtypes and which tensors the 5.0 GiB at 1B includes; it is consistent with two 2-byte moments for 1.34B parameters.

Our implementation uses BF16 for model parameters, gradients, and optimizer states. Specifically, the 5.0 GiB optimizer-state memory for the 1B Full-size model consists of AdamW's first- and second-moment tensors. Each tensor contains 1.339B elements stored in BF16 (2 bytes per element), giving a total of 2×1.339×109×2/230≈4.992 \times 1.339 \times 10^9 \times 2 / 2^{30} \approx 4.99 GiB. This matches the reviewer's calculation. We will clarify the tensor contents and data types in App. B.3.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer DBJB,

Thank you for your feedback on the benchmark recommendations and measurement settings. As the discussion period draws to a close, we wanted to follow up on the revised recommendations in Table 1 and the new throughput and checkpointing results. If you have a chance to review these updates, we would appreciate hearing whether they address your concerns and would be happy to clarify any remaining points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal revisions

Dear Reviewers, ACs, SACs, and PCs,

Thank you for your careful reviews and constructive feedback. We summarize the main revisions below.

Updated quality results and Tab. 1 recommendations

  • DBJB (W1 and Q3) and A567 (W1). We reran selected configurations and updated the FineWeb-Edu PPL results. Using AdamW with β1=0.9\beta_1=0.9 and β2=0.95\beta_2=0.95, Full-size PPL improved from 17.43 to 16.97 at 130M and from 13.40 to 13.15 at 350M (Tabs. 2 and 4–6). Tab. 1 now lists Fira, APOLLO, and Alice jointly at 350M under both matched-token and matched-FLOPs budgets. App. C.2 clarifies how paired runs inform the PPL tolerance.

Additional system profiling experiments

  • DBJB (W2 and Q1). We remeasured training throughput at 32,768 tokens per GPU per optimizer update and updated the wall-clock recommendations in Tab. 1 (Sec. 5.2, Tab. 11).
  • fPya (W1 and Q1). We added throughput comparisons at sequence lengths 256, 512, and 1024 and compared math SDPA and Flash SDPA at 350M (Tabs. 12–13).
  • DBJB (W2 and Q2). We added standard GCP measurements and remeasured structure-aware AC after correcting its implementation. With standard GCP at 1B, CoLA and LORO use 42–43% less peak reserved memory than Full-size (Tab. 1, App. D.3, Tab. 15).
  • A567 (W2, Q1, Q2, and Q5). We remeasured serving latency and peak memory with preallocated KV caches and BF16 reduction enabled for math SDPA. The updated results compare direct execution and V/O absorption under a shared serving protocol (Sec. 6, Fig. 4, Apps. B.5 and E, Tabs. 17–18).

Analysis and protocol clarifications

  • fPya (W2) and A567 (Q2). We clarified how V/O absorption uses the existing factors in low-rank weight checkpoints and quantified its trade-off between KV-cache storage and attention arithmetic (Sec. 6, App. E.2).
  • DBJB (Q2) and A567 (Q3). We specified that the 5.0 GiB of optimizer state in 1B Full-size consists of the two BF16 AdamW moment tensors (App. B.3).
  • A567 (Q4). We clarified how early-update profiling estimates cumulative training FLOPs (Sec. 3, App. B.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.