- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the effects of observation noise in Bayesian optimization (BO), this paper proposes RoSF-BO, a framework that generates augmented training samples intended to have reduced noise interference. Specifically, it applies clustering and residual-cancellation-based matching to the existing observations, followed by interpolation to construct synthetic samples. These generated samples are then combined with the original observations to train the surrogate model used in BO.
Critical strengths
- Overall, the paper is relatively clearly written. The algorithmic framework and workflow are presented clearly, making the proposed method reasonably easy to follow.
- The experimental evaluation is fairly comprehensive, and the paper also provides a theoretical analysis of the proposed method.
Critical weaknesses
- The paper has several noticeable writing and presentation issues. For example, after introducing the abbreviation Bayesian optimization (BO), the manuscript repeatedly uses the full term instead of consistently using the abbreviation. In addition, the citation style is inconsistent, with apparent misuse of
\citet,\citep, and\cite, making the references difficult to follow. There are also minor typographical or formatting issues, such as the appearance of an unexpected question mark. These issues should be carefully corrected. - The method seems to be essentially a data augmentation technique for noisy data, rather than one that is inherently tied to BO. However, this could also be an advantage.
Questions for authors
-
Applicability beyond BO: The core idea of this method seems to be data augmentation for noisy data, which could potentially be applied independently of the BO framework. I wonder whether it could also be effective when incorporated into other methods, such as neural network architectures. Of course, I'm just curious.
-
Computational complexity and runtime: Constructing the augmented dataset introduces additional computational overhead. Moreover, training the BO surrogate on the combined dataset may further increase the computational cost, particularly when the number of synthetic samples is large. However, the paper does not appear to provide a detailed complexity analysis or a sufficiently clear assessment of the resulting runtime overhead. Could the authors quantify the computational complexity and report the additional runtime introduced by RoSF-BO?
-
Presentation of the theoretical analysis: The Introduction identifies a theoretical analysis of RoSF-BO's effectiveness as the second main contribution, but the theoretical analysis is presented entirely in the appendix rather than in the main text. Could the authors explain this presentation choice and clarify the main theoretical findings and their implications in the main paper?
-
Sensitivity analysis of hyperparameters: In the parameter analysis, the paper considers only and . Is this range sufficient to establish the robustness of the method to these hyperparameters? Could the authors provide additional experiments or explain how these values were selected?
-
Choice of surrogate models: Why is the combination of GP + RoSF-BO not evaluated? Since RoSF-BO appears to be designed as an observation-level augmentation module that can potentially be combined with different surrogate models, evaluating it with a GP surrogate would help clarify whether its benefits are specific to TabPFN or generalize across surrogate-model choices.
-
Comparison with standard interpolation: The proposed method uses residual-based matching to construct interpolated samples. How much does this matching strategy contribute to the observed improvements compared with ordinary interpolation, such as randomly pairing observations and interpolating between them? An ablation study comparing residual-based matching with standard interpolation would help isolate the contribution of the proposed residual-cancellation mechanism.
5 comments
Q6: Comparison with Standard Interpolation
Thank you for this suggestion. We already include these comparisons in Appendices B.7 and B.8:
-
Interpolation baselines. Appendix B.7 compares RoSF-BO with random and nearest-neighbor interpolation. Both use the same linear interpolation procedure but different pairing strategies.
-
Fair comparison. We keep the number of pairs and generated samples equal to those of RoSF-BO, so the comparison focuses on pairing rather than augmentation size.
-
Sample quality. Figure 9 shows better overall MAE, WIA, and MAPE results for RoSF-BO against noise-free objectives, supporting the benefit of residual matching beyond interpolation alone.
-
BO performance. Appendix B.8 compares these strategies on six HPA problems with the same surrogate and multiple acquisition functions. RoSF-BO delivers more consistent gains than random or nearest-neighbor interpolation.
W1: Writing and Presentation Issues
These issues were an oversight on our part, and we appreciate you pointing them out. We have standardized the citation formatting, including the use of \cite, corrected the formatting issues throughout, and uploaded the revised manuscript. Thank you again for your careful review.
We hope our responses address your concerns, and we wish you all the best!
Acknowledged Efforts, but Fundamental Concerns on Methodological Scope and Efficiency Remain
I thank the authors for their detailed and careful responses, which have helped me better understand the content of the paper. However, I still maintain that the core contribution of this work lies in data expansion or data augmentation, rather than algorithmic innovation specifically for Bayesian Optimization (BO). Essentially, the method applies a general-purpose data processing technique to the observational data within BO. As such, this work might be more suitable for a venue focused on data augmentation or data mining.
Furthermore, as demonstrated in your computational complexity analysis, the introduction of the RoSF module significantly increases computational overhead . Since this process is embedded within the iterative loop of BO, this overhead is further compounded. While it is true that objective function evaluations can be expensive and time-consuming, we generally still aim to minimize the computational costs (including memory and time overheads) of the surrogate model and acquisition optimization. The substantial extra computational burden introduced by RoSF makes it less attractive for practical deployment, especially in scenarios where function evaluations are relatively cheap.
In summary, I believe the overall quality and novelty of this paper fall below the acceptance threshold of ICLR. Therefore, I will maintain my original score of Weak Rejection.
Q1/W2: Applicability beyond BO
Yes, RoSF can be used as an independent data-processing module beyond BO. We are happy to share two experiments we have already completed in other settings:
Small-sample prediction. Generating additional training samples using this augmentation approach substantially improves predictive performance on small datasets.
Image classification. In preliminary experiments on simple image-classification tasks, we augment image features extracted by a convolutional network and train only the fully connected classification head. This also yields substantial improvements in prediction accuracy.
We are also exploring augmentation of experience data for offline RL to improve learning efficiency, although this work is still at an early stage. If accepted, we will release our code on GitHub, where you can also follow our related work.
Q2: Computational complexity and runtime
Thank you for highlighting this point. RoSF-BO introduces computational overhead in both data augmentation and surrogate fitting on the enlarged dataset, while requiring no additional expensive objective evaluations.
1. Computational Complexity
Let n, D, and M denote the numbers of original observations, input dimensions, and generated samples, respectively. With K subspaces, the main costs are:
Surrogate fitting costs increase from T(n) to T(n+M). For a standard dense GP, this requires O((n+M)3).
2. Runtime Overhead
We measure runtime on a 10-dimensional Sphere problem with 100 initial observations, K=5, and η=5, using a GP surrogate. The augmented dataset contains approximately 599 samples. Results report the mean ± standard deviation over five runs on a single CPU thread. The additional runtime mainly comes from fitting the surrogate model on the enlarged training set.
Q3: Presentation of the theoretical analysis
Thank you for this suggestion.
Q4: Sensitivity Analysis of Hyperparameters
To address the limited parameter range, we extend the analysis to approximately K=2–50 and η=1–25 on six simulated datasets with different sample sizes, dimensions, feature distributions, and noise settings; see https://anonymous.4open.science/r/ICLR-27-1-2134/f1.pdf.
Sensitivity to K. With η=1, error reductions persist across a broad range of K, although the best choice varies across datasets.
Sensitivity to η. With K selected for each dataset, increasing η generally improves MAE before the gains plateau. MAPE shows more fluctuations, indicating some metric-dependent sensitivity.
Q5: Performance with Other Surrogate Models