- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a benchmark, coined LR-EPB, comparing ten low-rank pretraining methods and two dense references on LLaMA-style models from 60M to 1B parameters.
It evaluates model quality under matched-token and matched-FLOPs budgets and measures training memory, throughput, and serving latency.
The results show that lower FLOPs allow more training tokens and can narrow the quality gap, while actual memory savings and training speed depend on the method, model scale, and workload.
The paper also evaluates V/O absorption, which reuses trained low-rank factors to reduce value-cache storage and improve decoding speed at large batch sizes without retraining.
Critical strengths
1, The benchmark makes the effect of the training budget clear. Comparing the same methods under matched-token and matched-FLOPs budgets shows how additional training tokens can narrow the quality gap of low-rank models. This helps explain why evaluations using different budgets can reach different conclusions.
2, The paper measures whether theoretical savings translate into practical benefits. It reports peak memory and training throughput alongside FLOPs, showing that optimizer-state compression can provide limited memory savings when activations dominate, and that fewer FLOPs do not necessarily mean faster training.
3, The serving analysis provides a useful way to reuse the trained low-rank structure. V/O absorption preserves the model computation without retraining and reduces value-cache storage. The paper also identifies its workload dependence: the additional arithmetic can hurt smaller batches, while reduced cache traffic benefits larger batches.
Critical weaknesses
1, Each configuration is trained with one seed, and the methods use different tuning procedures and learning-rate schedules. The tolerance estimated from matched-token and matched-FLOPs runs does not establish variation across seeds, so small differences are insufficient to identify a clear winner. Repeating the key comparisons with comparable tuning budgets would make the method recommendations more convincing.
2, For the largest model at batch 128, a single BF16 KV cache should occupy about 15 GiB, while the reported direct-execution peak is 46.2 GiB. Temporary allocations may explain this difference, but their contribution is unclear. Similarly, the 2.1–2.2× decoding speedup cannot be explained by the 1.6× KV-cache reduction alone. A memory breakdown and kernel profile would help establish how much of the gain comes from absorption and how much depends on the implementation.
Questions for authors
1, For the largest model at batch 128 and context length 640, one BF16 KV cache should occupy about 15 GiB. Including the factorized weights gives approximately 16.1 GiB, compared with the reported peak of 46.2 GiB. Doubling the batch from 64 to 128 increases memory by 22.5 GiB, three times the expected cache increase. Could the authors specify the cache dtype and separate persistent storage from temporary allocations or retained copies?
2, V/O absorption reduces total KV-cache storage by a factor of 1.6, while the reported decoding speedup reaches 2.1–2.2× and value-aggregation arithmetic increases. These results are possible, but cache compression alone does not explain them. Could the authors provide kernel timings and memory-traffic measurements to identify the additional sources of improvement?
3, The reported optimizer-state memory of approximately 5 GiB for 1.339B parameters matches two BF16 moment tensors; two FP32 moment tensors would require approximately 10 GiB. Could the authors specify parameter, gradient, optimizer-state, and master-weight precision for each method? This detail affects the interpretation of the memory-saving results.
4, The main text describes FLOP measurement throughout training, while Appendix B.1 describes profiling early updates and extrapolating. Could the authors clarify how this accounts for projection refreshes, Riemannian steps, ReLoRA merges, and its initial dense-training phase? Reporting ordinary-step costs, periodic-operation costs, and their frequencies would make the accounting easier to verify.
5, For 350M LORO at batch 64, direct decoding takes 28.0 ms in Table 15 and 23.6 ms in Table 16, a difference of approximately 16%. The paper acknowledges separate profiling sessions, but this variation could affect comparisons near the latency crossover. Could the authors confirm identical execution settings and report variation across repeated measurements?
8 comments
Response to Reviewer A567: W2-part2, Q2 and Q3
W2-part2 and Q2. W2-part2: The 2.1–2.2× decoding speedup cannot be explained by the 1.6× KV-cache reduction alone. A memory breakdown and kernel profile would help establish how much of the gain comes from absorption and how much depends on the implementation. Q2: V/O absorption reduces total KV-cache storage by a factor of 1.6, while the reported decoding speedup reaches 2.1–2.2× and value-aggregation arithmetic increases. These results are possible, but cache compression alone does not explain them. Could the authors provide kernel timings and memory-traffic measurements to identify the additional sources of improvement?
We conduct additional kernel profiling and repeat the serving experiments to identify the sources of the decoding speedup. The results show that the original 2.1–2.2× speedup is partly inflated by two implementation factors.
First, the original direct execution uses default math SDPA, which converts the full BF16 K/V caches to FP32 during decoding. In contrast, V/O absorption uses BF16 matrix multiplications and FP32 softmax, avoiding these conversions. At 1B, batch 128, and context length 640, the 48 conversion kernels account for 41.36 ms per decoding step, corresponding to approximately 15 GiB of logical reads and 30 GiB of logical writes based on tensor sizes. With preallocated KV caches, enabling BF16 math reduction eliminates these conversions and reduces direct decoding latency from 92.82 to 30.98 ms per step. Absorbed execution takes 28.08 ms in the same single-step profile.
Second, dynamic KV-cache allocation also contributes to the original speedup. At 1B and batch 128, cache appending takes 29.97 ms for direct execution and 17.83 ms for absorption. With preallocated KV buffers, this overhead decreases to approximately 0.2 ms for both paths.
We therefore repeat all 53 serving configurations using preallocated BF16 KV caches and improved BF16 math settings. The following table reports the corrected time per output token (TPOT) in milliseconds.
| Scale / Batch | Model | Full-size | Direct | Absorbed | Speedup vs. Direct | Speedup vs. Full-size |
|---|---|---|---|---|---|---|
| 350M / 128 | CoLA | 20.37 | 26.40 | 25.89 | 1.02× | 0.79× |
| 350M / 128 | LORO | 20.37 | 31.26 | 30.06 | 1.04× | 0.68× |
| 350M / 256 | CoLA | 33.85 | 35.06 | 24.64 | 1.42× | 1.37× |
| 350M / 256 | LORO | 33.85 | 35.69 | 28.90 | 1.24× | 1.17× |
| 1B / 128 | CoLA | 38.85 | 37.36 | 25.83 | 1.45× | 1.50× |
| 1B / 128 | LORO | 38.85 | 37.98 | 30.26 | 1.26× | 1.28× |
The corrected results confirm that V/O absorption still provides substantial decoding speedups at larger batch sizes. At 1B and batch 128, CoLA and LORO achieve 1.50× and 1.28× the decoding speed of Full-size, respectively. At 350M, absorption becomes beneficial at batch 256, while it remains slower than Full-size at batch 128. These results support our finding that the benefit of V/O absorption depends on the trade-off between reduced KV-cache traffic and additional attention-value aggregation. We will update the serving results and clarify the execution settings in the revised paper.
Q3. The reported optimizer-state memory of approximately 5 GiB for 1.339B parameters matches two BF16 moment tensors; two FP32 moment tensors would require approximately 10 GiB. Could the authors specify parameter, gradient, optimizer-state, and master-weight precision for each method? This detail affects the interpretation of the memory-saving results.
Our implementation uses BF16 for model parameters, gradients, and persistent optimizer states across all twelve methods. SinkGD maintains no optimizer states, and no separate FP32 master-weight copies are maintained.
For the 1B Full-size model, the reported 5.0 GiB optimizer-state memory consists of AdamW's first- and second-moment tensors. Each tensor contains 1.339B BF16 elements, giving a total of GiB. Thus, the reported memory savings are measured under a consistent BF16 training configuration. We will explicitly document these precision settings in App. B.3.
Response to Reviewer A567: Q4 and Q5
Q4. The main text describes FLOP measurement throughout training, while Appendix B.1 describes profiling early updates and extrapolating. Could the authors clarify how this accounts for projection refreshes, Riemannian steps, ReLoRA merges, and its initial dense-training phase? Reporting ordinary-step costs, periodic-operation costs, and their frequencies would make the accounting easier to verify.
Our FLOP accounting includes both regular training updates and method-specific periodic operations. Specifically, we use PyTorch's FlopCounterMode to profile the first 500 optimizer updates, including forward and backward passes, optimizer steps, and periodic operations occurring within this window. We also register additional FLOP-counting formulas for SVD, QR factorization, and linear solves. For operations not captured by the counter, we include explicit FLOP corrections. The average FLOPs per update, including amortized periodic costs, is then used to track the cumulative training budget.
The periodic operations and their frequencies are summarized below.
| Method | Periodic operation | Frequency |
|---|---|---|
| GaLore, GoLore, Fira, Alice, APOLLO | Projection or basis refresh | Every 200 updates |
| LORO | Exact Riemannian step | Every 500 updates |
| ReLoRA | Adapter merging | Every 2,000 updates |
ReLoRA is handled separately because it contains different training phases. We calculate FLOPs for its initial dense-training phase and subsequent low-rank adapter phase separately, and include the cost of periodic adapter merges. The initial dense-training phase lasts 2,000, 4,000, and 10,000 updates at 60M, 130M, and 350M, respectively.
Thus, the throughout training description in Sec. 3 refers to tracking cumulative FLOPs throughout training, rather than continuously running the FLOP counter. We will clarify this distinction in Sec. 3 and App. B.1 and provide the ordinary-step and periodic-operation FLOP costs alongside their frequencies.
Q5. For 350M LORO at batch 64, direct decoding takes 28.0 ms in Table 15 and 23.6 ms in Table 16, a difference of approximately 16%. The paper acknowledges separate profiling sessions, but this variation could affect comparisons near the latency crossover. Could the authors confirm identical execution settings and report variation across repeated measurements?
We conduct a unified remeasurement of all 53 serving configurations to ensure consistent comparisons. The original direct-execution values of 28.0 and 23.6 ms are measured on different machines at different times. In the new experiments, we use the same physical NVIDIA A800, preallocated BF16 KV caches, and a consistent measurement protocol. Paired direct and absorbed runs also use identical model weights and fixed input token sequences.
For 350M LORO at batch 64, with a 512-token prompt and 128 generated tokens, we obtain the following results:
| Measurement | Direct TPOT (ms) | Absorbed TPOT (ms) |
|---|---|---|
| Original Tab. 15 | 28.0 | – |
| Original Tab. 16 | 23.6 | 26.5 |
| Unified remeasurement | 31.62 ± 0.55 | 28.78 ± 0.92 |
The updated values report mean ± standard deviation across 20 measured requests.
Under the unified protocol, V/O absorption achieves a 1.10× speedup over direct execution at this configuration. We will update Fig. 4 and Tabs. 15–16 using the same measurement campaign and revise the latency crossover analysis accordingly.
Response to Reviewer A567: W1, W2-part1 and Q1
Thank you for your thorough review and detailed technical comments. Your questions have been particularly helpful in refining our experimental analysis and measurements. We address each point below with additional results and clarifications. Please let us know if you have any further concerns.
We revise the method recommendations to avoid selecting a unique winner based on small PPL differences. We also recheck selected configurations with noticeably different PPL despite similar training budgets. The revised Full-size reference improves from 17.43 to 16.97 at 130M and from 13.40 to 13.15 at 350M. The updated comparisons are shown below.
At 350M, Fira, APOLLO, and Alice differ by at most 0.26 PPL under matched tokens and 0.13 under matched FLOPs. At 130M under matched tokens, Fira (16.98) is also nearly identical to Full-size (16.97). These differences do not support unique winner rankings.
Regarding tuning, Full-size uses a six-value learning-rate search at each model scale, while the other methods follow their authors' recommended configurations, with adjustments for training stability where needed. We report these settings in App. A.3–A.4. The 0.5-PPL tolerance is based on near-equivalent runs under the two budgets, rather than seed-to-seed variation. Therefore, we will update Tab. 1 to group methods with comparable quality and provide recommendations based on their measured memory and throughput trade-offs, rather than small differences in PPL.
We identify the source of the inflated memory usage and conduct additional serving measurements. In our original implementation, the KV cache grows through
torch.cat, which temporarily allocates new tensors while retaining the previous cache. Moreover, the previous request's final cache remains in memory until the current request finishes. Consequently, multiple cache-sized tensors can coexist, leading to an inflated peak. In a controlled experiment at batch 128, retaining the previous request's cache increases peak memory from 33.04 to 48.04 GiB, an additional 15.00 GiB.We correct this by preallocating BF16 K/V buffers and updating them in place. Each request reuses the buffers, and attention accesses only the valid cache prefix. We repeat all 53 serving configurations on the same NVIDIA A800 using the corrected cache implementation. The following table reports the results for 1B CoLA. All values are in GiB.
The corrected measurements substantially reduce the reported peak memory. At batch 128, direct execution now uses 22.41 GiB instead of 46.17 GiB. V/O absorption further reduces the peak to 17.41 GiB, while the KV-cache storage decreases from 14.98 to 9.36 GiB. The remaining memory beyond the KV cache includes model weights and temporary allocations.
We will update the serving memory results and clarify the cache allocation and measurement protocol in the revised paper.