- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents an application of Bayesian optimization (BO) to optimize the parallel training throughput. Since the parallel training is not completely black-box, we can exploit the problem structure to improve the sample efficiency of BO specifically for this application. The authors proposed to exploit the domain knowledge for GP modeling. The experiments showed that the proposed method successfully identifies good configurations.
Critical strengths
- the paper tries to address an important application
- the experiments seem to be executed in large scale
Critical weaknesses
The authors should first enhance the presentation quality. The current manuscript is not ready for novelty discussion. The paper maybe presents something novel, but it is hard for me to know how the method is different from existing approaches. Essentially, it is very common to insert the mean and model the residual part only by GP and to use early termination. Therefore, the proposed pipeline looks to me an assemblance of well known promising techniques. If the authors, on the contrary, argue this, please report how experts would usually combine existing techniques and how such combinations fail. I understood that the authors are proposing something, but it is not presented in a way that people know why naively applying existing methods does not work well and how the proposed method circumvents these problems.
The manuscript needs significant revision
The paper is not ready for publication. Please do NOT assume that reviewers will read appendices. They should be supplementary, and we cannot judge the paper correctness if the manuscript heavily depends on appendices given the limited review period. Such paper organization is the authors' responsibility, so defending your manuscript by saying "Look at Appendix X" would not be constructive. I will list the reasons.
First, the design choice is not meaningfully discussed. For example, informed GP prior belief and early trial termination are already known to perform nicely in practice, so my impression on Fig. 7 saying the combination works better than only either of them is no surprise to me. The real question is what people would not come up with easily and why or how it contributes to the performance. Maybe, the way the authors apply GP prior belief or the early trial termination is novel, but no experiments against naive applications are found and I am not sure how exactly the proposed combination is different from a simple combination.
Furthermore, the space is not used well. It is nice to have many figures, but they are mostly very small because some parts are redundant. The background should concisely explain dp, tp, pp. I don't think we need an almost full page for this basic concept considering the tightly packed method and experiment sections. Plus, Fig. 2 is not effectively used. dp, tp, pp could be compacted significantly if Fig. 2 is organized more visually. With the saved space, the relationship to multi-fidelity BO should also be discussed in the main paper. For example, BOHB [1] is likely a good baseline method in this application.
Additionally, there are so many italic and bold characters, making it hard for me to focus which ones are really important. Although it is fine to use italic and bold, their usage should be more focused for readability.
The theoretical analysis assumes i.i.d. Gaussian observations and derives a particular . However, the authors set the variable as fixed. Why? Doesn't this invalidate the theoretical guarantee for the claimed method?
References
- [1] Falkner, Stefan, Aaron Klein, and Frank Hutter. "BOHB: Robust and efficient hyperparameter optimization at scale." International conference on machine learning. PMLR, 2018.
Questions for authors
Questions
- Please see
Critical Weaknesses.
Suggestions
- Please see
Critical Weaknesses.
4 comments
Response (Part 2/3)
First, the design choice is not meaningfully discussed. For example, informed GP prior belief and early trial termination are already known to perform nicely in practice, so my impression on Fig. 7 saying the combination works better than only either of them is no surprise to me...
We note that Fig. 7 shows results from our ablation studies to isolate the effect of each proposed efficiency trick on OPPA's performance (Sec. 5), rather than to claim that their combination is surprising. As discussed in our response to the first comment, while inserting a prior mean is a known idea, the form of the prior belief suitable in the case of PC optimization is not clear (Sec. 4.1). Similarly, while techniques similar to our early termination mechanism have previously been studied in other BO contexts, they cannot be naively applied for PC optimization as they are either inefficient or ill-suited for our problem setting, as we have elaborated in App. H. Here, we specifically can note the following:
-
The measurements that are made are repeated noisy measurements of the same quantity. Therefore, the measured throughput is still used when a trial terminates early, with the variance of the measured throughput being higher due to fewer training steps. This is unlike a multi-fidelity setting where the observations are not repeated noisy measurements of the same quantity, but instead only correlated across different fidelity values.
-
Our method performs termination during the NN training trial, and proposes a cheap method to determine when this can be done, as the indicator in Eq. 2 only compares the throughput estimate of the current trial with that of the best PC found so far. That is, the early termination mechanism is designed while taking into consideration the characteristics of training trials, which consist of sequential training steps and can be reduced during the actual training run via a quick decision process.
-
To save time, a trial should be terminated as early as possible once we know the PC is unlikely to be optimal. In practice, we see that the first few training steps are insufficient to decide this due to warm-up effects, and so OPPA always runs at least steps before deciding whether to terminate the trial (as seen in the condition in Eq. 2). We show that even when we adaptively terminate the training trial early, we are still able to guarantee a regret bound which is sublinear and dependent on (Theorem 4.1).
Please do NOT assume that reviewers will read appendices. They should be supplementary, and we cannot judge the paper correctness if the manuscript heavily depends on appendices given the limited review period...
We thank the reviewer for this comment. We would like to clarify that the main paper already carefully explains the important points of our work, including the problem formulation (Sec. 3), the design of each component of OPPA (Sec. 4), our main theoretical result (Thm. 4.1) and the experimental results (Sec. 5), and only defers to the appendix for further details (e.g., full proofs and additional experimental results), with appropriate signposts to the relevant sections. This is necessary given the space limitations for the submission (9 pages for the main text), and we hope the reviewer understands. Nonetheless, in the updated paper, we have used the additional page to move more of the important discussion points from the appendix to the main paper, including the comparison of the early termination mechanism with existing methods (from App. H), the choice of the hyperparameters of OPPA (e.g., and , from App. G), and the rationale for our choice of baselines and how the best PC is reported (from App. I).
The background should concisely explain dp, tp, pp...
We note that in our paper, we provide a more extensive description (with diagrams) for different types of parallelism as we do not assume the readers are necessarily practitioners who work with parallel NN training already. As we discuss in Sec. 2 (around Line 158), PC optimization differs from typical BO applications since the hyperparameters in a PC affecting the training throughput have better-defined mechanics (even if not completely known) that can be partially described based on domain knowledge, which OPPA exploits through its prior belief (Sec. 4.1). Hence, our paper also aims to be a starting point for BO researchers who may want to consider PC optimization as a novel setting, and to develop techniques that exploit its specific problem structure. Since this requires more knowledge of how each type of parallelism affects the training throughput and memory usage, we have decided to provide more extensive background information in the main paper, while keeping further details in App. A.
Response (Part 3/3)
The theoretical analysis assumes i.i.d. Gaussian observations and derives a particular . However, the authors set the variable as fixed. Why?
The theoretical choice of in Thm. F.2 (Eq. 51) depends on the noise standard deviation , which is unknown in practice. We therefore fix to a constant (as stated in Sec. 4.3 of the updated paper). Setting to a small enough number (in our case, ) is sufficient to distinguish between suboptimal PCs. We note that this is in line with other existing BO works [6,7] which typically use fixed constants in place of theoretically optimal values which are difficult to compute in practice, while still maintaining good empirical results. We also see that the choice of does not affect the sublinear regret in Thm. F.1, meaning the algorithm will still recover the optimum in theory.
We again sincerely thank the reviewer for their efforts during the review process, and hope our clarifications are able to address your concerns and improve your evaluation of our paper.
[1] Picheny et al. Quantile-Based Optimization of Noisy Computer Experiments With Tunable Precision. Technometrics 2013.
[2] Binois et al. Replication or Exploration? Sequential Design for Stochastic Simulation Experiments. Technometrics 2019.
[3] Dai et al. Batch Bayesian Optimization for Replicable Experimental Design. NeurIPS 2023.
[4] Ishibashi et al. A Stopping Criterion for Bayesian Optimization by the Gap of Expected Minimum Simple Regrets. AISTATS 2023.
[5] Wilson. Stopping Bayesian Optimization with Probabilistic Regret Bounds. NeurIPS 2024.
[6] Bogunovic et al. Adversarially Robust Optimization with Gaussian Processes. NeurIPS 2018.
[7] Li & Scarlett. Gaussian Process Bandit Optimization with Few Batches. AISTATS 2022.
Thank you for the comments. Please also check the original review.
it is not obvious what form of the prior belief is suitable for PC optimization the main paper already carefully explains the important points of our work we provide a more extensive description (with diagrams) for different types of parallelism as we do not assume the readers are necessarily practitioners who work with parallel NN training already
Probably, the authors explain important points carefully, but the amount isn't correctly assigned. Importantly, the main body has only 9 pages (& 10 pages during the rebuttal period), so more important points should be more carefully discussed.
About Sec. 4.1, although describes the design (prior decomposition, GP for deviation, R and M forms), and it gives a generic reason for the overall structure, it does not discuss why the specific prior forms were chosen or what alternatives were rejected. If the choice is not obvious, please do explain the rationale in the main body with more space. That is what I said more important.
the early termination mechanism is novel in that it is able to exploit characteristics of PC optimization in a way that has not been previously considered in other BO works.
Same for this part. It's clear where the decision is made and what is measured, but I don't think the delta is informatively discussed in the manuscript. The closest prior works are successive halving and Hyperband, median stopping, and confidence-bound elimination but the authors seem to define Repeated-measurement BO and BO stopping criteria as the closest related work. Essentially, benchmarking is not performed against strong existing baselines such as BOHB or learning-curve extrapolation or freeze-thaw BO. The design choices inside the rule again get little justification.
We note that Fig. 7 shows results from our ablation studies to isolate the effect of each proposed efficiency trick on OPPA's performance (Sec. 5)
The ablation is not meaningful without alternative approaches if the authors really believe that the proposed choices are novel. As mentioned, we already know that each component such as GP mean insertion does work in practice. The whole point is how to build the pipeline. I don't think the pipeline design rationale is fully explained in the main body; hence, the current ablation study is also not self-contained in my eyes.
Overall, my assessment does not change. I strongly recommend major revision.
Response (Part 1/3)
We thank the reviewer for their review, and their comments regarding how we study a problem that is practically relevant, and how our experiments are executed at large scale. Below, we provide a response to the comments raised by the reviewer.
We note that one of the contributions in our work is adapting BO to parallelism configuration (PC) optimization which, as noted by the reviewer, is an important problem in itself and is worth studying. In this regard, we are able to introduce several modifications to vanilla BO which are specific to the problem of optimizing PCs (e.g., parallelism-informed prior belief, early termination), which together improve the performance of OPPA beyond vanilla BO. We note that the points below are already discussed in the paper, and we indicate where for each of them.
First, while inserting a prior mean is a known idea, realizing it for PC optimization is not straightforward, since it is not obvious what form of the prior belief is suitable for PC optimization. In OPPA, we presented a prior belief function which is able to sufficiently capture the trends within the parallelized training process, while also being general enough to work across a wide range of NNs and hardware setups (see Sec. 4.1). This demonstrates how combining appropriate domain knowledge with real measurements is useful for PC optimization.
Second, the early termination mechanism is novel in that it is able to exploit characteristics of PC optimization in a way that has not been previously considered in other BO works. Existing BO works which use repeated measurements of the same input [1,2,3] fix the number of repetitions up front. This is unsuitable for PC optimization since initializing NN training is costly and rather that it is more efficient to decide the number of training steps on-the-fly during the training loop rather than restarting trials. Meanwhile, stopping criteria for BO [4,5] decide when to terminate the BO loop itself, which is a different stage of the optimization, and are too expensive to run repeatedly during NN training. In contrast, the early termination mechanism exploits the sequential nature of NN training and incurs little overhead. We note that this discussion was already included in App. H. However, an abridged version of this discussion has also been moved to the main text in the last paragraph of Sec. 4.3.