acceptodds
Under review as a conference paper at ICLR 2027

SmartBCD: Smart Block Coordinate Descent via Cost-Aware Scheduling

Abstract

Full-parameter fine-tuning (FFT) of large language models requires substantial GPU memory. Block coordinate descent (BCD)-based methods address this challenge through block-wise parameter updates rather than simultaneous updates to all parameters, enabling more memory-efficient FFT. However, existing BCD-based methods either follow predetermined or randomized block schedules, or prioritize blocks primarily based on optimization signals, without explicitly accounting for the joint variation in optimization gain and computation cost across training stages. This can lead to suboptimal use of limited training resources. We propose SmartBCD, a cost-aware block scheduling method that improves resource utilization. SmartBCD estimates each block’s utility as the predicted loss reduction per unit of training time and uses this estimate to schedule updates as training progresses. A maximum-wait constraint ensures that every block continues to receive updates. We conduct experiments with TinyLlama-1.1B and Qwen2.5-7B on instruction-following and mathematical reasoning tasks. On our Qwen2.5-7B MathInstruct setting, under the same optimization-step budget, SmartBCD reduces end-to-end training time by 4.4% and final validation loss by 1.3% relative to BAdam-Random, without negligible additional GPU-memory overhead.

Primary area:
optimization
Keywords:
Large language models, Full-parameter fine-tuning, Block coordinate descent, Cost-aware block scheduling, Memory-efficient training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uUrbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Memory-efficient full-parameter fine-tuning can update one parameter block at a time, making the choice of block important. SmartBCD extends BAdam with a scheduler that ranks blocks by predicted training-loss reduction per unit of measured time, while enforcing a maximum wait before each block is revisited. The reported experiments show modest improvements in training time and validation loss over random and cyclic scheduling.

Critical strengths
  1. The scheduling objective is intuitive, and the maximum-wait rule gives a clear block-coverage guarantee.
  2. The paper measures wall-clock time and energy, includes a Gain-only control, and evaluates two model sizes and two datasets.
Critical weaknesses
  1. The main 49,038-step results are single runs; only the shorter 896-step experiments use three seeds. This leaves uncertainty around the reported 4.4% time and 1.3% validation-loss improvements.
  2. The evidence for cost normalization itself is limited: on the primary setting, Gain-only reaches the same final validation loss, takes only 1.5% longer, and has a slightly higher average downstream score (Table 3). The ablations also show small, inconsistent effects for several scheduler components (Table 4).
  3. The paper discusses adaptive block selection in BlockLLM but does not compare against it. This limits the claim that SmartBCD improves on existing adaptive scheduling methods.
Questions for authors
  1. Can the authors report paired-seed results for the full K=100 runs, especially SmartBCD versus Gain-only, with uncertainty for time, loss, energy, and downstream scores?
  2. How does SmartBCD compare with an implemented adaptive block-selection baseline such as BlockLLM under matched blocks and budgets?
  3. Why does peak reserved memory differ substantially among block methods in Table 7 despite their shared block partition?
1 comment
AuthorsOpenReview ↗

Response to Reviewer uUrb

Q1. Paired-seed reliability of full-scale experiments

We thank the reviewer for identifying the limitations of the original single-run full-scale evaluation.

To provide stronger evidence, we added full-scale Qwen2.5-7B experiments with three independent random seeds for SmartBCD and Gain-only, using the 49,038-step training budget. The results are summarized in Figure 5 and Table 13.

SmartBCD achieves an average training time of 11.912 ± 0.034 hours, compared with 12.238 ± 0.039 hours for Gain-only, representing approximately 2.66% lower mean runtime. Their final validation losses remain close, at 0.2271 ± 0.0003 and 0.2270 ± 0.0003, respectively.

We additionally report the mean and standard deviation of downstream benchmark scores, rather than relying solely on single-run averages. These results also reveal performance trade-offs, including lower GSM8K accuracy for SmartBCD.

We acknowledge that the current multi-seed results do not establish statistical reliability for every comparison in the original paper. In particular, the complete multi-seed evaluation does not yet include BAdam-Random, and the reported multi-seed results do not include energy variability.

We therefore interpret the new results as evidence of consistent runtime improvement over Gain-only in the evaluated runs, rather than as proof that every reported improvement is statistically significant across all baselines and metrics.

Q2. Comparison against an implemented adaptive baseline

We agree that a comparison against an adaptive scheduling baseline is necessary to distinguish cost-aware scheduling from adaptive selection in general.

To address this concern, we added a BlockLLM-adapted baseline in Section 4.5 and Appendix E.5, evaluated with the same 28-block partition and optimization-step budget.

Across three independent full-scale runs, the BlockLLM-adapted baseline achieves 12.278 ± 0.041 hours of training time and 0.2295 ± 0.0004 final validation loss. SmartBCD achieves 11.912 ± 0.034 hours and 0.2271 ± 0.0003, respectively. These results demonstrate an advantage for SmartBCD in the joint comparison of training time and terminal validation loss under the evaluated setup.

At the same time, BlockLLM-adapted obtains higher average downstream accuracy. We therefore do not claim that SmartBCD universally outperforms BlockLLM across all evaluation metrics.

We will also clarify how BlockLLM was adapted to our common block-wise training framework, so that the comparison can be interpreted in light of the implementation differences rather than as an unconditional comparison with the original BlockLLM setup.

Q3. Differences in peak reserved GPU memory

We thank the reviewer for pointing out the discrepancy between peak allocated and peak reserved GPU memory.

In the primary Qwen2.5-7B MathInstruct experiment, the block-coordinate methods have similar peak allocated memory: 30,339 MiB for BAdam-Random, 30,354 MiB for BAdam-Cyclic, and 30,339 MiB for SmartBCD. However, their peak reserved memory differs substantially: 31,072 MiB, 41,340 MiB, and 31,094 MiB, respectively, as reported in Table 8.

These metrics represent different quantities in PyTorch. Peak allocated memory tracks memory occupied by tensors, whereas peak reserved memory includes memory managed by the CUDA caching allocator, including cached allocations that may not currently be occupied by tensors.

Consequently, similar allocated-memory peaks do not necessarily imply similar reserved-memory peaks. Differences in block-update sequences may alter allocation and reuse patterns, leading to different caching allocator behavior and potentially different degrees of fragmentation.

Importantly, SmartBCD's peak reserved memory differs by only approximately 0.07% from BAdam-Random in the primary experiment, suggesting that its efficiency advantage does not require a measurable increase in peak allocated memory or a substantial increase in reserved memory relative to that baseline.

Reviewer_KuRFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SmartBCD, a cost-aware block scheduling method for memory-efficient full-parameter fine-tuning of LLMs. Building on BAdam, SmartBCD estimates each parameter block's optimization gain using a first-order approximation based on gradients and parameter updates, and normalizes this gain by measured training time. The scheduler combines this utility with exploration and recency terms, while enforcing a maximum-wait constraint to prevent block starvation.

Experiments on TinyLlama-1.1B and Qwen2.5-7B across MathInstruct and Alpaca-GPT4 demonstrate improvements over random and cyclic block scheduling. In the primary Qwen2.5-7B MathInstruct experiment, SmartBCD reduces training time by 4.4% and final validation loss by 1.3% relative to BAdam-Random, with negligible additional memory overhead. The paper also provides ablations, fixed-time comparisons, downstream evaluations, and additional hardware experiments.

Critical strengths
  1. Practical and well-motivated scheduling objective. The paper identifies an important limitation of existing block scheduling methods: optimization gain alone does not necessarily reflect computational efficiency. Incorporating measured training cost provides a sensible approach to improving resource allocation. The method is straightforward to integrate into existing BCD training frameworks without additional forward or backward passes for block selection.

  2. Broad experimental evaluation with useful controlled comparisons. The evaluation considers multiple models, datasets, scheduling baselines, and resource metrics. In particular, the Gain-only baseline helps isolate the contribution of cost normalization, while fixed-time comparisons and component ablations provide additional insights into the scheduling mechanism. The consistent improvements over random and cyclic scheduling suggest practical utility.

Critical weaknesses
  1. Limited incremental benefit attributable to cost awareness. Although SmartBCD improves over random and cyclic scheduling, its advantage over the Gain-only variant is relatively small. On Qwen2.5-7B MathInstruct, both methods achieve the same final validation loss (0.2275), while SmartBCD reduces training time from 12.091 to 11.908 hours, an improvement of approximately 1.5%. Gain-only also achieves a slightly higher average downstream score (41.75 vs. 41.40). These results suggest that much of the improvement over conventional BCD may come from adaptive gain-based selection rather than cost normalization itself. Furthermore, the lack of an experimental comparison with closely related adaptive scheduling methods, particularly BlockLLM, makes it difficult to assess the contribution relative to prior work.

  2. Insufficient statistical evidence for the reported improvements. The main full-scale experiments appear to use a single run per configuration, while the three-seed analyses are restricted to substantially shorter 896-step experiments with a different block-epoch length. Given the relatively small performance differences, especially against Gain-only, it is difficult to determine whether the gains are robust to training randomness and runtime variability. Moreover, some individual downstream results are noticeably worse: SmartBCD achieves 59.06 on GSM8K, compared with 67.95 for Gain-only and 66.10 for BAdam-Cyclic. The reported averages alone do not establish consistent downstream performance.

Questions for authors

Q1. Statistical reliability. Could the authors report results from multiple independent full-scale runs, particularly comparing SmartBCD with Gain-only? Reporting mean and variance for training time, validation loss, and downstream performance would help establish whether the approximately 1.5% runtime improvement is statistically reliable.

Q2. Comparison with adaptive scheduling baselines. How does SmartBCD compare with BlockLLM or other adaptive importance-based block selection methods under matched training budgets? Such comparisons would help distinguish the benefits of cost awareness from those of adaptive scheduling more generally.

Q3. Scheduling overhead. Could the authors quantify the additional computation and memory required to calculate the first-order gain term, including the gradient–parameter-update inner products? This would help clarify the net cost of the scheduling mechanism, especially given its modest advantage over Gain-only.

3 comments
AuthorsOpenReview ↗

Response to Reviewer KuRF

Q1. Statistical reliability of the reported improvements

We thank the reviewer for emphasizing the need to distinguish improvements caused by cost-aware scheduling from variations due to training randomness.

To strengthen the statistical evidence, we extended our full-scale Qwen2.5-7B MathInstruct evaluation to three independent random seeds. The new results are reported in Section 4.5 (Figure 5) and Appendix E.5 (Table 13). Across these runs, SmartBCD achieves an average training time of 11.912 ± 0.034 hours, compared with 12.238 ± 0.039 hours for Gain-only. This corresponds to a 2.66% reduction in mean training time.

The final validation losses are similar: 0.2271 ± 0.0003 for SmartBCD and 0.2270 ± 0.0003 for Gain-only. These results suggest that the runtime advantage is maintained across the evaluated random seeds without a substantial change in terminal validation loss. We also report downstream performance variability. SmartBCD obtains an average score of 41.93 ± 0.60, compared with 42.21 ± 0.45 for Gain-only. We acknowledge that SmartBCD does not consistently outperform Gain-only on individual downstream tasks, particularly GSM8K.

Accordingly, we focus our claim on computational efficiency rather than consistent downstream accuracy improvements. The new experiments provide additional evidence of runtime robustness.

Q2. Comparison with BlockLLM and adaptive scheduling baselines

We appreciate this suggestion and agree that comparison with adaptive scheduling methods is important for isolating the contribution of cost awareness.

We have added a BlockLLM-adapted baseline to our full-scale Qwen2.5-7B MathInstruct experiments, using the same block partition and optimizer-step budget. The results are presented in Figure 5 and Table 13.

Across three independent runs, the BlockLLM-adapted baseline requires 12.278 ± 0.041 hours and achieves a final validation loss of 0.2295 ± 0.0004. In comparison, SmartBCD requires 11.912 ± 0.034 hours and achieves 0.2271 ± 0.0003. Thus, SmartBCD reduces the mean training time by approximately 2.98% while obtaining a lower final validation loss in this experimental setting.

However, BlockLLM-adapted achieves a higher average downstream score (43.87 ± 0.65 versus 41.93 ± 0.60), indicating that SmartBCD does not dominate every evaluation dimension.These results support the value of cost-aware block scheduling for training-time efficiency and validation-loss optimization.

Q3. Computational and memory overhead of scheduling

We thank the reviewer for requesting a quantitative analysis of the first-order gain calculation.

We have added a controlled profiling study in Appendix B.4. To isolate the gain calculation overhead, we fixed the block-selection sequence to a deterministic cyclic order and compared runs with and without computing the first-order term (−gᵀΔθ). In a 280-step CUDA micro-benchmark covering all 28 blocks, the additional latency was 4.836 ms per optimizer step at the 95th percentile.

In a separate 2,800-step macro-benchmark, the trainer execution times were 2848.267 seconds with gain computation and 2840.714 seconds without it. The measured difference is 7.55 seconds, corresponding to approximately 0.27% additional runtime. The two configurations also showed no measured difference in peak reserved GPU memory, while their validation-loss trajectories matched to machine precision.

The gain calculation operates on the active block and does not require additional forward or backward passes for evaluating inactive blocks. The subsequent block-selection logic requires only (O(D)) scalar operations and state.

Importantly, Gain-only computes the same first-order gain term as SmartBCD. Thus, the measured gain-calculation overhead is shared by both methods and does not explain the observed runtime advantage of SmartBCD over Gain-only.These measurements indicate that the additional cost of obtaining the scheduling signal is small relative to the overall training workload, although it is not mathematically zero.

Thank you for the additional multi-seed results and the BlockLLM comparison. While the runtime advantage over Gain-only is now clearer (approximately 2.7%), the absolute improvement remains modest, and SmartBCD does not show consistent gains on downstream tasks. The new evidence addresses the statistical reliability concern but does not substantially change my assessment of the practical significance of cost-aware scheduling. I will maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer KuRF: On Practical Significance and Algorithmic Efficiency

Dear Reviewer KuRF,

Thank you for your prompt response and for acknowledging that our new multi-seed experiments successfully address the statistical reliability concerns. We deeply appreciate your continued engagement. Regarding the practical significance of the improvements, we would respectfully like to offer a brief concluding perspective on the algorithmic value of SmartBCD, which we hope the Area Chair and the reviewers will consider:

  1. The "Free Lunch" of Algorithmic Elegance (The 2.7% Gain): While a 2.7% absolute runtime reduction over the Gain-only baseline might appear numerically modest at first glance, its algorithmic significance is profound. Both SmartBCD and Gain-only compute the exact same first-order metric. The 2.7% wall-clock acceleration is achieved purely by mathematically rearranging the block update sequence, without skipping any necessary gradient steps or introducing any computational overhead (as proven by our 0.27% profiling overhead). In systems optimization, squeezing an additional 2.7% of physical execution time out of an already highly optimized, metric-matched baseline—simply through smarter scheduling—is a non-trivial algorithmic "free lunch."
  2. Validation Loss vs. Downstream Variance: Regarding downstream tasks, we agree that the gains are not uniform across every specific benchmark. However, as is well-established in optimization literature, discrete downstream tasks are subject to high variance and domain-specific overfitting (e.g., BlockLLM scoring higher on GSM8K but suffering the worst general validation loss). By explicitly incorporating hardware costs, SmartBCD robustly achieves the lowest continuous Validation Loss in the absolute shortest physical time. This guarantees that the foundation model becomes a better generalist, which is the ultimate goal of a principled optimization scheduler. We thank you again for pushing us to strengthen the empirical rigor of this work. The multi-seed and profiling results you inspired have made the final manuscript substantially more complete.
Reviewer_mwPbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SmartBCD reformulates block scheduling as a dynamic resource allocation problem: for each parameter block, it considers both the optimization gain derived from its update and the wall-clock training cost required to complete that update. It then prioritizes blocks that offer the greatest optimization gain per unit of training time.

Critical strengths
  1. Different blocks yield varying optimization gains and may also require different training durations. Scheduling based solely on gain or a fixed sequence often fails to make the most efficient use of a limited training budget. By reframing block selection as a resource allocation problem—specifically, maximizing "optimization gain per unit of training time"—the approach becomes logically intuitive and directly aligned with actual system costs.
  2. SmartBCD does not require a re-evaluation of all blocks at every scheduling decision point.
Critical weaknesses
  1. The largest model evaluated is only 7B, and the training settings are quite limited. In particular, the heterogeneity of block training costs is likely influenced by factors such as sequence length, activation checkpointing, and the use of Mixture-of-Experts (MoE) architectures.

  2. There is a lack of comprehensive sensitivity analysis regarding the scheduler’s hyperparameters.

  3. Given that this is a "cost-aware" scheduling approach, the performance improvement—reducing training time from 12.458 hours to 11.908 hours compared to a random baseline—is relatively modest.

Questions for authors

See "Weaknesses" for further details.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mwPb

Q1. Limited model scale and heterogeneous training conditions

We thank the reviewer for highlighting the importance of evaluating block scheduling under heterogeneous training conditions. We agree that sequence length, activation checkpointing, and model architecture significantly affect block-level computation costs. In our current Qwen2.5-7B experiments, gradient checkpointing is enabled (as specified in Table 6). Consequently, the observed variation in block execution times—where block-epoch durations naturally range from 60s to over 110s across different blocks (Figure 2-left)—already reflects a training environment incorporating checkpointing, although our experiments do not separately isolate its contribution to cost heterogeneity.

Our scheduling mechanism uses the measured execution cost of previously updated blocks rather than assuming uniform block costs. This design allows the scheduler to respond to observed cost differences without requiring a predefined analytical cost model. Nevertheless, we acknowledge that our current evaluation is limited to dense models up to 7B parameters, with a fixed sequence-length configuration. We have therefore expanded the Conclusion and Discussion section to explicitly identify MoE architectures, variable sequence lengths, and alternative checkpointing configurations as important directions for further evaluation.

We emphasize that whether increased cost heterogeneity leads to larger scheduling benefits remains an empirical question, rather than a conclusion established by our current experiments.

Q2. Hyperparameter sensitivity

We thank the reviewer for this suggestion. To address the concern, we added a hyperparameter sensitivity study in Section 4.5 (Table 4), using the controlled 896-step setting with three independent random seeds. Specifically, we vary the exploration coefficient c (0.01, 0.10, 0.50) and the discounted-count decay factor rho (0.90, 0.97, 0.99) , while keeping the remaining scheduler settings fixed.

Across these configurations, SmartBCD achieves an average optimization efficiency ranging from 0.87222 × 10⁻³ to 0.88555 × 10⁻³, compared with 0.83677 × 10⁻³ for BAdam-Random. The corresponding training times range from 874.38 to 888.90 seconds, compared with 927.59 seconds for Random.

We also observe that Random achieves a slightly lower terminal validation loss under the same optimizer-step budget in this short-run setting. Thus, the sensitivity results support the stability of SmartBCD's optimization efficiency per unit of computation time, rather than uniform superiority in terminal validation loss.

These results indicate that the observed efficiency advantage is maintained across the tested values of (c) and (rho). We acknowledge that this is a targeted sensitivity study, not an exhaustive exploration of all scheduler hyperparameters.

Q3. Modest end-to-end performance improvement

We agree that the 4.4% end-to-end runtime reduction relative to BAdam-Random is numerically modest, and we appreciate the opportunity to clarify its practical significance.

First, the improvement is obtained under the same optimizer-step budget while also reducing final validation loss from 0.2305 to 0.2275. Under the fixed 11.908-hour wall-clock budget in Table 2, SmartBCD achieves a validation loss of 0.2274, compared with 0.2305 for Random and 0.2288 for Cyclic.

Second, the benefit of cost normalization is more pronounced at certain intermediate training stages. As shown in Appendix E.2 (Table 9), over the validation-loss targets from 0.275 to 0.240, SmartBCD requires on average 11.7% less wall-clock time and 14.6% less GPU energy than Gain-only. These figures describe the measured first-passage trajectories, and the advantage becomes smaller near terminal convergence.

Third, the additional computation is small relative to the training workload. Our controlled profiling study in Appendix B.4 measures a 0.27% macro-level runtime overhead from the first-order gain calculation, without additional model forward or backward passes for block selection.

We therefore position SmartBCD as an incremental but measurable improvement in computation allocation. Its contribution is not a dramatic reduction in individual-step complexity, but a more effective use of the existing block-wise optimization budget.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kind Reminder: Follow-up on Our Rebuttal

Dear Reviewer mwPb, Reviewer KuRF, and Reviewer uUrb,

We hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, we wanted to follow up to ensure we have addressed all of your concerns satisfactorily. If you feel that our rebuttal has adequately addressed your concerns, we would be very grateful if you would consider raising your score. If there are any additional points or feedback you would like us to consider, please let us know. Your insights are invaluable to us, and we are eager to address any remaining issues to improve our work.

Thank you very much for your time and effort in reviewing our paper.

Best regards,

The authors of Paper 57257

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.