- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Memory-efficient full-parameter fine-tuning can update one parameter block at a time, making the choice of block important. SmartBCD extends BAdam with a scheduler that ranks blocks by predicted training-loss reduction per unit of measured time, while enforcing a maximum wait before each block is revisited. The reported experiments show modest improvements in training time and validation loss over random and cyclic scheduling.
Critical strengths
- The scheduling objective is intuitive, and the maximum-wait rule gives a clear block-coverage guarantee.
- The paper measures wall-clock time and energy, includes a Gain-only control, and evaluates two model sizes and two datasets.
Critical weaknesses
- The main 49,038-step results are single runs; only the shorter 896-step experiments use three seeds. This leaves uncertainty around the reported 4.4% time and 1.3% validation-loss improvements.
- The evidence for cost normalization itself is limited: on the primary setting, Gain-only reaches the same final validation loss, takes only 1.5% longer, and has a slightly higher average downstream score (Table 3). The ablations also show small, inconsistent effects for several scheduler components (Table 4).
- The paper discusses adaptive block selection in BlockLLM but does not compare against it. This limits the claim that SmartBCD improves on existing adaptive scheduling methods.
Questions for authors
- Can the authors report paired-seed results for the full K=100 runs, especially SmartBCD versus Gain-only, with uncertainty for time, loss, energy, and downstream scores?
- How does SmartBCD compare with an implemented adaptive block-selection baseline such as BlockLLM under matched blocks and budgets?
- Why does peak reserved memory differ substantially among block methods in Table 7 despite their shared block partition?
Response to Reviewer uUrb
Q1. Paired-seed reliability of full-scale experiments
We thank the reviewer for identifying the limitations of the original single-run full-scale evaluation.
To provide stronger evidence, we added full-scale Qwen2.5-7B experiments with three independent random seeds for SmartBCD and Gain-only, using the 49,038-step training budget. The results are summarized in Figure 5 and Table 13.
SmartBCD achieves an average training time of 11.912 ± 0.034 hours, compared with 12.238 ± 0.039 hours for Gain-only, representing approximately 2.66% lower mean runtime. Their final validation losses remain close, at 0.2271 ± 0.0003 and 0.2270 ± 0.0003, respectively.
We additionally report the mean and standard deviation of downstream benchmark scores, rather than relying solely on single-run averages. These results also reveal performance trade-offs, including lower GSM8K accuracy for SmartBCD.
We acknowledge that the current multi-seed results do not establish statistical reliability for every comparison in the original paper. In particular, the complete multi-seed evaluation does not yet include BAdam-Random, and the reported multi-seed results do not include energy variability.
We therefore interpret the new results as evidence of consistent runtime improvement over Gain-only in the evaluated runs, rather than as proof that every reported improvement is statistically significant across all baselines and metrics.
Q2. Comparison against an implemented adaptive baseline
We agree that a comparison against an adaptive scheduling baseline is necessary to distinguish cost-aware scheduling from adaptive selection in general.
To address this concern, we added a BlockLLM-adapted baseline in Section 4.5 and Appendix E.5, evaluated with the same 28-block partition and optimization-step budget.
Across three independent full-scale runs, the BlockLLM-adapted baseline achieves 12.278 ± 0.041 hours of training time and 0.2295 ± 0.0004 final validation loss. SmartBCD achieves 11.912 ± 0.034 hours and 0.2271 ± 0.0003, respectively. These results demonstrate an advantage for SmartBCD in the joint comparison of training time and terminal validation loss under the evaluated setup.
At the same time, BlockLLM-adapted obtains higher average downstream accuracy. We therefore do not claim that SmartBCD universally outperforms BlockLLM across all evaluation metrics.
We will also clarify how BlockLLM was adapted to our common block-wise training framework, so that the comparison can be interpreted in light of the implementation differences rather than as an unconditional comparison with the original BlockLLM setup.
Q3. Differences in peak reserved GPU memory
We thank the reviewer for pointing out the discrepancy between peak allocated and peak reserved GPU memory.
In the primary Qwen2.5-7B MathInstruct experiment, the block-coordinate methods have similar peak allocated memory: 30,339 MiB for BAdam-Random, 30,354 MiB for BAdam-Cyclic, and 30,339 MiB for SmartBCD. However, their peak reserved memory differs substantially: 31,072 MiB, 41,340 MiB, and 31,094 MiB, respectively, as reported in Table 8.
These metrics represent different quantities in PyTorch. Peak allocated memory tracks memory occupied by tensors, whereas peak reserved memory includes memory managed by the CUDA caching allocator, including cached allocations that may not currently be occupied by tensors.
Consequently, similar allocated-memory peaks do not necessarily imply similar reserved-memory peaks. Differences in block-update sequences may alter allocation and reuse patterns, leading to different caching allocator behavior and potentially different degrees of fragmentation.
Importantly, SmartBCD's peak reserved memory differs by only approximately 0.07% from BAdam-Random in the primary experiment, suggesting that its efficiency advantage does not require a measurable increase in peak allocated memory or a substantial increase in reserved memory relative to that baseline.