acceptodds
Under review as a conference paper at ICLR 2027

RoSF-BO: Robust Surrogate Fitting via Low-Noise Data Augmentation for Bayesian Optimization

Abstract

Bayesian optimization (BO) relies on surrogate models fitted to limited observations, making its performance particularly sensitive to observation noise. Existing noise-handling approaches mainly focus on surrogate modeling or acquisition design, while observation-level noise mitigation remains relatively underexplored. To address this issue, we propose RoSF-BO, an observation-level data augmentation framework that improves the training data used for surrogate fitting without additional expensive evaluations or discarding original observations. RoSF-BO exploits local functional relationships among evaluated solutions to identify observations with complementary deviations and generates additional training samples with reduced noise interference through residual-cancellation interpolation. We further provide a theoretical analysis to explain the noise-reduction effect of the generated samples. Extensive experiments on static datasets, noisy single-objective BO, and noisy multi-objective BO demonstrate that RoSF-BO consistently improves data quality and translates these improvements into better optimization performance across different BO methods and observation distributions. Overall, RoSF-BO provides a new observation-level perspective for noise handling in Bayesian optimization and complements existing surrogate- and acquisition-level approaches while preserving the standard BO acquisition and evaluation procedures.

Primary area:
optimization
Keywords:
Bayesian optimization, Data Augmentation, Robust Surrogate Fitting, Expensive Black-Box Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JdqVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the effects of observation noise in Bayesian optimization (BO), this paper proposes RoSF-BO, a framework that generates augmented training samples intended to have reduced noise interference. Specifically, it applies clustering and residual-cancellation-based matching to the existing observations, followed by interpolation to construct synthetic samples. These generated samples are then combined with the original observations to train the surrogate model used in BO.

Critical strengths
  1. Overall, the paper is relatively clearly written. The algorithmic framework and workflow are presented clearly, making the proposed method reasonably easy to follow.
  2. The experimental evaluation is fairly comprehensive, and the paper also provides a theoretical analysis of the proposed method.
Critical weaknesses
  1. The paper has several noticeable writing and presentation issues. For example, after introducing the abbreviation Bayesian optimization (BO), the manuscript repeatedly uses the full term instead of consistently using the abbreviation. In addition, the citation style is inconsistent, with apparent misuse of \citet, \citep, and \cite, making the references difficult to follow. There are also minor typographical or formatting issues, such as the appearance of an unexpected question mark. These issues should be carefully corrected.
  2. The method seems to be essentially a data augmentation technique for noisy data, rather than one that is inherently tied to BO. However, this could also be an advantage.
Questions for authors
  1. Applicability beyond BO: The core idea of this method seems to be data augmentation for noisy data, which could potentially be applied independently of the BO framework. I wonder whether it could also be effective when incorporated into other methods, such as neural network architectures. Of course, I'm just curious.

  2. Computational complexity and runtime: Constructing the augmented dataset DRoSF-BOD_{\mathrm{RoSF\text{-}BO}} introduces additional computational overhead. Moreover, training the BO surrogate on the combined dataset Dt∪DRoSF-BOD_t \cup D_{\mathrm{RoSF\text{-}BO}} may further increase the computational cost, particularly when the number of synthetic samples is large. However, the paper does not appear to provide a detailed complexity analysis or a sufficiently clear assessment of the resulting runtime overhead. Could the authors quantify the computational complexity and report the additional runtime introduced by RoSF-BO?

  3. Presentation of the theoretical analysis: The Introduction identifies a theoretical analysis of RoSF-BO's effectiveness as the second main contribution, but the theoretical analysis is presented entirely in the appendix rather than in the main text. Could the authors explain this presentation choice and clarify the main theoretical findings and their implications in the main paper?

  4. Sensitivity analysis of hyperparameters: In the parameter analysis, the paper considers only K∈{5,10,15}K \in \{5,10,15\} and η∈{1,5}\eta \in \{1,5\}. Is this range sufficient to establish the robustness of the method to these hyperparameters? Could the authors provide additional experiments or explain how these values were selected?

  5. Choice of surrogate models: Why is the combination of GP + RoSF-BO not evaluated? Since RoSF-BO appears to be designed as an observation-level augmentation module that can potentially be combined with different surrogate models, evaluating it with a GP surrogate would help clarify whether its benefits are specific to TabPFN or generalize across surrogate-model choices.

  6. Comparison with standard interpolation: The proposed method uses residual-based matching to construct interpolated samples. How much does this matching strategy contribute to the observed improvements compared with ordinary interpolation, such as randomly pairing observations and interpolating between them? An ablation study comparing residual-based matching with standard interpolation would help isolate the contribution of the proposed residual-cancellation mechanism.

5 comments

Q1/W2: Applicability beyond BO

Yes, RoSF can be used as an independent data-processing module beyond BO. We are happy to share two experiments we have already completed in other settings:

  1. Small-sample prediction. Generating additional training samples using this augmentation approach substantially improves predictive performance on small datasets.

  2. Image classification. In preliminary experiments on simple image-classification tasks, we augment image features extracted by a convolutional network and train only the fully connected classification head. This also yields substantial improvements in prediction accuracy.

We are also exploring augmentation of experience data for offline RL to improve learning efficiency, although this work is still at an early stage. If accepted, we will release our code on GitHub, where you can also follow our related work.


Q2: Computational complexity and runtime

Thank you for highlighting this point. RoSF-BO introduces computational overhead in both data augmentation and surrogate fitting on the enlarged dataset, while requiring no additional expensive objective evaluations.

1. Computational Complexity

Let nn, DD, and MM denote the numbers of original observations, input dimensions, and generated samples, respectively. With KK subspaces, the main costs are:

  • Evolutionary clustering: O(PG nKD)O(PG\,nKD), where PP and GG are the population size and number of generations.
  • TSP ordering: O(K2D+IK2)O(K^2D+IK^2), where II is the number of 2-opt passes.
  • Local regression: O(nD2+KD3)O(nD^2+KD^3) using dense least squares.
  • Residual matching: approximately O(Hn2/K+n3/K2)O(Hn^2/K+n^3/K^2) for roughly balanced subspaces, including cost construction and Hungarian matching. Here, HH is the number of interpolation positions used to evaluate residual cancellation.
  • Sample generation: O(MD)O(MD).

Surrogate fitting costs increase from T(n)T(n) to T(n+M)T(n+M). For a standard dense GP, this requires O((n+M)3)O((n+M)^3).

2. Runtime Overhead

We measure runtime on a 10-dimensional Sphere problem with 100 initial observations, K=5K=5, and η=5\eta=5, using a GP surrogate. The augmented dataset contains approximately 599 samples. Results report the mean ± standard deviation over five runs on a single CPU thread. The additional runtime mainly comes from fitting the surrogate model on the enlarged training set.

ComponentWithout augmentation (ms)With augmentation (ms)
Clustering—36.40 ± 3.51
TSP ordering—0.10 ± 0.01
Local regression—0.91 ± 0.09
Residual matching—0.43 ± 0.06
Interpolation and other processing—1.72
Total augmentation039.83 ± 3.65
GP fitting20.97 ± 1.95466.92 ± 60.08
Total20.97509.85

Q3: Presentation of the theoretical analysis

Thank you for this suggestion.

  • Given the page limit and the equations and discussion involved in the theoretical derivations, we placed the full analysis in the appendix to keep the presentation of the method and experiments clear and coherent.
  • In the revised manuscript, we will summarize the main theoretical findings in the main text.

Q4: Sensitivity Analysis of Hyperparameters

To address the limited parameter range, we extend the analysis to approximately K=2K=2–5050 and η=1\eta=1–2525 on six simulated datasets with different sample sizes, dimensions, feature distributions, and noise settings; see https://anonymous.4open.science/r/ICLR-27-1-2134/f1.pdf.

  1. Sensitivity to KK. With η=1\eta=1, error reductions persist across a broad range of KK, although the best choice varies across datasets.

  2. Sensitivity to η\eta. With KK selected for each dataset, increasing η\eta generally improves MAE before the gains plateau. MAPE shows more fluctuations, indicating some metric-dependent sensitivity.


Q5: Performance with Other Surrogate Models

  1. Compatibility with other surrogates. To directly address your concern, we evaluate RoSF-BO with GP and Random Forest (RF) surrogates to assess its sensitivity to surrogate choice. See https://anonymous.4open.science/r/ICLR-2027-2-B919/rosf_gp_rf_combined.pdf.
  2. Broad effectiveness across surrogates and BO methods. With both GP and RF surrogates, RoSF-BO delivers gains across multiple BO methods in single- and multi-objective settings, demonstrating that its benefits extend beyond TabPFN and supporting its broad applicability.

Q6: Comparison with Standard Interpolation

Thank you for this suggestion. We already include these comparisons in Appendices B.7 and B.8:

  1. Interpolation baselines. Appendix B.7 compares RoSF-BO with random and nearest-neighbor interpolation. Both use the same linear interpolation procedure but different pairing strategies.

  2. Fair comparison. We keep the number of pairs and generated samples equal to those of RoSF-BO, so the comparison focuses on pairing rather than augmentation size.

  3. Sample quality. Figure 9 shows better overall MAE, WIA, and MAPE results for RoSF-BO against noise-free objectives, supporting the benefit of residual matching beyond interpolation alone.

  4. BO performance. Appendix B.8 compares these strategies on six HPA problems with the same surrogate and multiple acquisition functions. RoSF-BO delivers more consistent gains than random or nearest-neighbor interpolation.


W1: Writing and Presentation Issues

These issues were an oversight on our part, and we appreciate you pointing them out. We have standardized the citation formatting, including the use of \cite, corrected the formatting issues throughout, and uploaded the revised manuscript. Thank you again for your careful review.


We hope our responses address your concerns, and we wish you all the best!

Reviewer_JdqVOpenReview ↗

Acknowledged Efforts, but Fundamental Concerns on Methodological Scope and Efficiency Remain

I thank the authors for their detailed and careful responses, which have helped me better understand the content of the paper. However, I still maintain that the core contribution of this work lies in data expansion or data augmentation, rather than algorithmic innovation specifically for Bayesian Optimization (BO). Essentially, the method applies a general-purpose data processing technique to the observational data within BO. As such, this work might be more suitable for a venue focused on data augmentation or data mining.

Furthermore, as demonstrated in your computational complexity analysis, the introduction of the RoSF module significantly increases computational overhead . Since this process is embedded within the iterative loop of BO, this overhead is further compounded. While it is true that objective function evaluations can be expensive and time-consuming, we generally still aim to minimize the computational costs (including memory and time overheads) of the surrogate model and acquisition optimization. The substantial extra computational burden introduced by RoSF makes it less attractive for practical deployment, especially in scenarios where function evaluations are relatively cheap.

In summary, I believe the overall quality and novelty of this paper fall below the acceptance threshold of ICLR. Therefore, I will maintain my original score of Weak Rejection.

Reviewer_467hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes RoSF-BO, a data augmentation framework for noisy Bayesian optimization. It generates synthetic training samples from existing observations without requiring additional expensive evaluations, aiming to improve surrogate fitting and subsequent acquisition decisions. The method first partitions the observed inputs into spatial clusters using an evolutionary clustering strategy, then orders these clusters by approximately solving a traveling salesperson problem (TSP). For each pair of neighboring clusters, a local regression model, linear regression in the experiments, is fitted to compute prediction residuals. Observations from the two clusters are then paired using a matching criterion that favors residual cancellation across interpolation positions. Each matched pair is linearly interpolated in both input x and output y space to generate synthetic samples, which are combined with the original observations for surrogate fitting. The authors provide a theoretical analysis relating noise reduction to endpoint noise cancellation and local interpolation error. They empirically evaluate the framework on static noisy datasets and single- and multi-objective BO problems, using TabPFN as the surrogate with several acquisition functions.

Critical strengths
  • The paper explores an interesting approach to noisy optimization by generating synthetic samples for surrogate training without additional expensive evaluations.
  • The writing is clear and easy to follow.
  • The empirical results show consistent improvements over the reported baselines, suggesting the approach has potential across the evaluated settings.
Critical weaknesses
  1. Insufficient justification of design choices. Several components lack a clear rationale, particularly the evolutionary clustering strategy and TSP-based cluster ordering. Although the ablations compare RoSF-BO with alternative pairing strategies, they do not isolate the contributions of these components or establish their advantages over simpler alternatives, such as K-means clustering or selecting neighboring clusters directly by distance.

  2. Limited evidence of applicability across surrogate models. The BO experiments demonstrate improvements when RoSF-BO is combined with TabPFN, but deliberately exclude GP + RoSF-BO. The authors justify this choice by mentioning dependencies introduced by augmentation, yet it is not clear how those dependencies affect GP-based BO in practice. Moreover, if the method’s benefits are restricted to a particular surrogate model, this would narrow its applicability and weaken the claimed generality of the contribution.

  3. Dependence on a large initial design is unexplored. The main experiments use 100 initial LHS evaluations followed by ~40–100 additional evaluations, with similarly large initial designs in the supplementary experiments. These settings provide substantial space-filling data for augmentation process. With a smaller initial design, local regression may be less reliable. Moreover, small initial design can make subsequent BO evaluations concentrate in certain regions, potentially affecting clustering and residual-based matching. Evaluating smaller initial designs under the same total evaluation budget would clarify the method’s effectiveness when expensive evaluations are particularly scarce.

  4. Observations for local linear regression. Related to the previous concern, a smaller initial design may have too few observations within neighboring-cluster pairs to support reliable local regression. A linear model in d dimensions has d+1 coefficients and requires at least d+1 observations for reliable estimation. Insufficient observations may therefore compromise the residuals used for matching. The authors should clarify how such cases are handled, especially for BO settings where observations are expensive and limited.

  5. Citation and presentation issues. Nearly all citations are awkwardly integrated into sentences (without proper parentheses), and the benchmark description contains an unresolved citation marked “?”. These issues give the manuscript an unfinished appearance

Questions for authors
  1. Could the authors evaluate different initial design sizes under the same total evaluation budget? This would clarify whether RoSF-BO remains effective with fewer initial observations and a more concentrated sampling distribution during subsequent optimization.
  2. Could the authors evaluate RoSF-BO with a GP surrogate, such as qEI + GP vs qEI + GP + RoSF-BO? This would help establish whether the benefits extend beyond TabPFN.
  3. Can RoSF-BO remain effective under other noise settings? Current experiments use a fixed heterogeneous noise scaled with the objective value.
8 comments

Thank you for taking the time to review our paper so carefully. We really appreciate your thoughtful suggestions and the opportunity to improve our work.


Q1/W3/W4: Performance with Small Initial Sample Sets

We understand your concern about whether RoSF-BO can still provide improvements with fewer initial observations.

1. Effect of Initial Sample Size

  • Our initial design of 10d10d follows a commonly used rule of thumb in Bayesian optimization, where dd is the input dimension [1,2].
  • We additionally evaluate smaller initial designs of 5d5d and 7.5d7.5d, keeping the total evaluation budget unchanged. As shown in https://anonymous.4open.science/r/ICLR-2027-2-B919/Sphere-D10.pdf.
    • Optimization performance: RoSF-BO maintains strong performance with fewer initial observations.
    • Sequential sampling distribution: We provide sampling heatmaps to visualize distribution differences and calculate the Average Nearest-Neighbor Distance (ANND) to quantify sample dispersion. Together, these results show that RoSF-BO explores the search space more thoroughly.

2. How Do We Handle Insufficient Samples for Local Regression? (W4)

  • We have considered this issue. Each local model uses all observations from two neighboring subspaces, so insufficient samples were rarely encountered in our experiments.
  • We add a constraint to the evolutionary search: each subspace must contain at least ⌈(d+1)/2⌉\lceil(d+1)/2\rceil observations, ensuring at least d+1d+1 observations per local fit. This is also one reason for choosing evolutionary clustering. Thank you for the suggestion; we will clarify this constraint in the manuscript.

Q2/W2: Performance with Other Surrogate Models

  1. Compatibility with other surrogates. To directly address your concern, we evaluate RoSF-BO with GP and Random Forest (RF) surrogates to assess its sensitivity to surrogate choice. See https://anonymous.4open.science/r/ICLR-2027-2-B919/rosf_gp_rf_combined.pdf.
  2. Broad effectiveness across surrogates and BO methods. With both GP and RF surrogates, RoSF-BO delivers gains across multiple BO methods in single- and multi-objective settings, demonstrating that its benefits extend beyond TabPFN and supporting its broad applicability.

Q3: Performance under Other Noise Settings

To address this concern, we evaluate six additional simulated datasets with varying noise mixtures, sample sizes, dimensions, and feature distributions. As shown in https://anonymous.4open.science/r/ICLR-27-1-2134/f1.pdf, error reductions persist across a broad range of KK and η\eta, supporting the effectiveness of RoSF-BO’s augmentation under these additional noise settings.


W1: Rationale for Evolutionary Clustering and TSP-Based Ordering

  1. K-means can be sensitive to its initial centroids. Evolutionary clustering explores multiple representative sets, reducing reliance on a single initialization. It also allows us to constrain the minimum sample count per subspace, addressing insufficient observations for local regression (see W4). We adopt it for this flexibility and to seek more stable performance.
  2. TSP-based ordering is important because it forms a closed tour covering all subspaces, with each subspace connected to two neighbors for pairing. In contrast, independently selecting nearest neighbors may produce repeated connections or disconnected groups of subspaces. As shown in https://anonymous.4open.science/r/ICLR-2027-2-B919/TSP_vs_NNS.pdf.

W5: Writing and Presentation Issues

These issues were an oversight on our part, and we appreciate you pointing them out. We have standardized the citation formatting, including the use of \cite, corrected the formatting issues throughout, and uploaded the revised manuscript. Thank you again for your careful review.


We hope our responses address your concerns, and we wish you all the best!


  • [1] Wu H, et al. A multistage expensive constrained multiobjective optimization algorithm based on ensemble infill criterion[J]. IEEE Transactions on Evolutionary Computation, 2024
  • [2] Borowska A, et al. Bayesian optimisation for efficient parameter inference in a cardiac mechanics model of the left ventricle[J]. International Journal for Numerical Methods in Biomedical Engineering, 2022

Thanks the authors for the response. I need to clarify some information in the rebuttal before I can consider it further.

  • Since "each subspace must contain at least ⌈(d+1)/2⌉\lceil (d+1)/2 \rceil observations, the required minimum number of initial data points for the proposed method to work (to have enough data points for local regression) is K⌈(d+1)/2⌉K\lceil (d+1)/2 \rceil. Am I right?
  • Can you explain more how to read f1.pdf, the "Performance under Other Noise Settings" experiment. I except to see similar figures like Figures 4,5, where methods are tested with and without RoSF-BO, under different noise settings.
  • In "rosf_gp_rf_combined.pdf", why are there no standard error bars? Did you run with only 1 seed?

Q1: Minimum Sample Requirement

  1. We apologize for not explaining this implementation detail clearly in our previous response.

  2. More specifically, the minimum subspace size is a soft constraint in our implementation: clustering solutions with fewer than ⌈(d+1)/2⌉\lceil(d+1)/2\rceil observations in a subspace receive a large penalty rather than being strictly excluded. Therefore, n0≥K⌈d+12⌉n_0 \ge K\left\lceil\frac{d+1}{2}\right\rceil is not a strict requirement for the method to operate.

  3. In our implementation, we check whether a neighboring-subspace pair contains more than d+2d+2 observations. Otherwise, we supplement the fitting set with the nearest observations from other subspaces until this threshold is met.

    • Why use d+2d+2 rather than d+1d+1? With d+1d+1 observations, a linear model fits the observations exactly, producing zero residuals that cannot indicate noise signs. We therefore require more than d+2d+2 observations to help obtain informative residuals for noise-sign identification.
  4. In practice, the preferred minimum size is satisfied in the vast majority of cases, since KK need not be large. This is also supported by the best-performing settings in https://anonymous.4open.science/r/ICLR-27-1-2134/f1.pdf. Please see Q2 for a detailed explanation of this experiment.

  5. Moreover, local regression need not recover the true function precisely; its purpose is to identify noise signs as reliably as possible and guide complementary pairing. Please see our response to Reviewer Wc5w under Q1/W1: Reliability of Local Regression Residuals for further discussion.

Q2

1. Purpose and interpretation of https://anonymous.4open.science/r/ICLR-27-1-2134/f1.pdf

  • This experiment evaluates how RoSF-BO improves noisy data on static datasets under different mixed-noise settings. It extends the experiments in Figures 2 and 3 of the manuscript.

  • The six datasets vary in noise composition, sample size, dimensionality, and input distribution. We measure changes in MAE and MAPE against the noise-free reference; negative values indicate error reductions relative to the original noisy data.

  • The upper panels vary KK while fixing η=1\eta=1, and the lower panels vary η\eta using the best-performing KK for each dataset. This lets us examine both hyperparameter sensitivity and error reduction across different noise configurations.

  • The results show improvements across a broad range of settings. This is a data-level evaluation of noise reduction.

2. BO performance under different noise settings

  • Our original experiments adopt objective-value-dependent noise scaling to accommodate substantial differences in objective magnitudes across benchmarks. For instance, objective values in our DTLZ3 and DTLZ7 configurations are approximately of orders 101010^{10} and 10110^1, respectively. Consequently, a common absolute noise magnitude would induce markedly different signal-to-noise ratios across problems, limiting the comparability of the resulting evaluations.
  • To address your concern, we additionally evaluate a noise setting in which the noise standard deviation is fixed at 15% of each objective’s range, following the range-based scaling in [1,2]. The results in https://anonymous.4open.science/r/ICLR-2027-3223/singleobjective-regret-and-multiobjective-hv.pdf, it shows that RoSF-BO continues to improve optimization performance under this setting.

Q3: Random Seeds and Standard Error Bars


  • [1] Daulton S, Balandat M, Bakshy E. Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement[C]. Advances in Neural Information Processing Systems, 2021, 34: 2187–2200.
  • [2] Picheny V, Wagner T, Ginsbourger D. A benchmark of kriging-based infill criteria for noisy optimization[J]. Structural and Multidisciplinary Optimization, 2013, 48(3): 607–626.
Reviewer_Wc5wOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoSF-BO, a data augmentation approach for noisy Bayesian optimization. It identifies pairs of observations with complementary residuals and generates additional training samples through interpolation. The method aims to reduce noise without additional objective evaluations. Experiments on synthetic and engineering benchmarks show improvements in single-objective and multi-objective optimization.

Critical strengths

The paper explores an interesting direction for handling noise by improving the training data rather than modifying the surrogate model or acquisition function. The experiments cover a reasonable range of benchmarks and show promising improvements. I also appreciate the comparisons with alternative data augmentation strategies in the appendix.

Critical weaknesses

My main concern is whether the residuals from local linear regression can reliably represent observation noise. For nonlinear functions, these residuals may also reflect modelling errors. This is particularly concerning in higher dimensions, where the number of observations in each local region may be insufficient for fitting a reliable linear model.

The theoretical analysis is somewhat weak. It shows that interpolation can reduce noise under certain conditions, but does not establish whether the proposed matching strategy satisfies these conditions. Interpolation may introduce substantial errors when paired observations are far apart, or the underlying function is highly nonlinear.

Finally, the main BO experiments combine RoSF-BO with TabPFN only, making it difficult to assess its generality across surrogate models.

Questions for authors

How do you ensure that local regression residuals reflect observation noise rather than model fitting errors, especially in higher dimensions?

Can you evaluate the actual errors of generated samples against the noise-free objective, particularly as the distance between paired observations increases?

How does RoSF-BO perform with other surrogate models, such as Gaussian processes?

How sensitive is the method to different noise distributions, including biased and heavy-tailed noise?

2 comments

Q1/W1: Reliability of Local Regression Residuals

We understand the reviewer’s concern. Local regression residuals indeed contain both model fitting error and observation noise:

\qquad d_r(x_i)=\hat f_r(x_i)-f(x_i).$$ Here, $d_r(x_i)$ is the local model error and $\epsilon_i$ is the observation noise. However, the local regression need not precisely recover the true function; its role is to distinguish observations with opposite noise signs as reliably as possible, enabling residual-based pairing to promote noise cancellation. We discuss two cases below. ### 1. Local model fits well - When the local fitting error is smaller in magnitude than the observation noise, residuals can reliably indicate noise signs, allowing RoSF-BO to **pair observations with opposite noise signs and promote cancellation through interpolation**, as shown in **https://anonymous.4open.science/r/ICLR-27-1-2134/Residual-guided%20pair.pdf**. ### 2. Local model fits less accurately - Even if the fit is poor, it can still be useful as long as it identifies noise signs correctly, which is easier than accurately fitting the true function. **This motivates our use of linear regression as a simple local approximation**, consistent with the established effectiveness of local linear modeling for smooth regression functions [1]. - Even if noise signs are misidentified and observations with same-sign noise are paired, the **interpolated noise component remains bounded**: $$|(1-\tau)\epsilon_i+\tau\epsilon_j| \le \max ( |\epsilon_i|,|\epsilon_j| ).$$ Thus, incorrect sign identification may weaken noise cancellation, but does not itself amplify noise. As illustrated in **https://anonymous.4open.science/r/ICLR-27-1-2134/Imperfect_residual_signs.pdf**. - Even in **high-dimensional settings**, identifying whether noise is positive or negative does not require an accurate fit of the true function. The augmentation results in **Appendix B.4** provide indirect empirical support for this explanation. **Overall, local regression is designed to guide complementary pairing, rather than precisely estimate noise. Even if noise signs are misidentified, interpolation does not amplify the endpoint-noise component beyond the larger endpoint magnitude.** ---- ## Q2/W2: Generated-Sample Errors and Pairing Distance 1. Our existing experiments partially address this question. **Appendix B.4** shows consistent error reductions against noise-free objectives in 10, 20, and 30 dimensions. As Euclidean distances generally grow with dimensionality , this provides indirect support for effectiveness at larger distances. We further summarize these results in plots to clarify the dimensionality effect: **https://anonymous.4open.science/r/ICLR-27-1-2134/mae_D10_D20_D30_combined.pdf**. 2. RoSF-BO encourages local pairing by partitioning observations into compact subspaces and matching only between neighboring subspaces. Thus, the method is explicitly designed to avoid distant pairs where possible and reduce the risk of long-distance interpolation. 3. To further address your concern, we use the number of clusters $K$ as an indirect proxy for pairing distance: decreasing $K$ generally enlarges subspaces and may increase the distances between paired observations. We evaluate six simulated datasets under different values of $K$ and $\eta$ ( `this experiment also helps address Q4, as discussed below`). As shown in **https://anonymous.4open.science/r/ICLR-27-1-2134/f1.pdf**, noise-related error metrics improve across a broad range of settings, including small $K$, providing evidence of robustness to increased pairing distances. ---- ## Q3/W3: Performance with Other Surrogate Models 1. **Choice of TabPFN.** We initially used GP surrogates and obtained good results, but GP fitting was slower than TabPFN in our implementation. 2. **Compatibility with other surrogates.** To directly address your concern, we evaluate RoSF-BO with GP and Random Forest (RF) surrogates to assess its sensitivity to surrogate choice. See **https://anonymous.4open.science/r/ICLR-2027-2-B919/rosf_gp_rf_combined.pdf**. 3. With both GP and RF surrogates, RoSF-BO delivers gains across multiple BO methods in single- and multi-objective settings, demonstrating that its benefits extend beyond TabPFN and supporting its broad applicability. ---- ## Q4: Sensitivity to Different Noise Distributions The experiments presented in `Q2, Point 3` also address this concern. We construct six simulated datasets with varying noise mixtures, sample sizes, dimensions, and feature distributions. Across a broad range of $K$ and $\eta$, the method consistently reduces noise-related errors. ---- **We thank the reviewer for the constructive comments. We hope our responses address your concerns, and we wish you all the best!** ---- - [1] Fan J. Design-adaptive nonparametric regression[J]. Journal of the American statistical Association, 1992

Thank you. The rebuttal has addressed the majority of my concerns. So I have upgraded my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses

We thank the reviewers for their thoughtful comments. In our responses, we have:

  1. Clarified residual-based pairing: We explained the roles of observation noise and local fitting error, illustrated accurate and imperfect noise-sign identification, and clarified the bound on the interpolated endpoint-noise component. (Reviewer Wc5w: Q1/W1)

  2. Expanded the data-quality and sensitivity analysis: We summarized the existing 10-, 20-, and 30-dimensional results and evaluated six additional simulated datasets across wider ranges of KK and η\eta, with varying noise mixtures and data characteristics. (Reviewer Wc5w: Q2/W2, Q4; Reviewer 467h: Q3; Reviewer JdqV: Q4)

  3. Extended evaluation beyond TabPFN: We evaluated GP and random forest surrogates, showing gains across multiple BO methods in both single- and multi-objective settings. (Reviewer Wc5w: Q3/W3; Reviewer 467h: Q2/W2; Reviewer JdqV: Q5)

  4. Examined smaller initial sample sets: We tested 5D5D and 7.5D7.5D initial designs under unchanged total evaluation budgets, and examined sequential sampling using heatmaps and average nearest-neighbor distances. We also clarified the minimum cluster-size constraint for local regression. (Reviewer 467h: Q1/W3/W4)

  5. Explained the clustering and ordering choices: We clarified how evolutionary clustering supports compact subspaces and sample-count constraints, and how TSP ordering provides a closed pairing structure. (Reviewer 467h: W1)

  6. Analyzed computational cost: We provided component-wise complexity analysis and runtime measurements, distinguishing augmentation overhead from surrogate-fitting costs. (Reviewer JdqV: Q2)

  7. Discussed broader applications and existing ablations: We shared experiments on small-sample prediction and image classification, and highlighted the controlled comparisons with random and nearest-neighbor interpolation already included in Appendices B.7–B.8. (Reviewer JdqV: Q1/W2, Q6)


The results and illustrations provided in our responses are summarized below.

ItemPurposeLink
Residual-guided pairingIllustrate noise cancellation under accurate local fittingIllustration
Imperfect residual signsIllustrate bounded endpoint-noise components when sign identification failsIllustration
Results across dimensionsSummarize data-quality improvements in 10, 20, and 30 dimensionsResults
Six simulated datasetsExamine wider hyperparameter ranges and different noise and data settingsResults
GP and RF surrogatesEvaluate performance beyond TabPFN in single- and multi-objective BOResults
Smaller initial designsCompare optimization performance and sampling distributions under a fixed total budgetResults
TSP versus nearest-neighbor selectionIllustrate the difference in subspace pairing structuresIllustration
Complexity and runtimeQuantify augmentation and surrogate-fitting costsReviewer JdqV: Q2
Standard interpolation comparisonsIsolate the benefit of residual-based pairingAppendices B.7–B.8

We also thank the reviewers for pointing out the formatting issues. We have corrected them and uploaded the revised manuscript.

We hope these responses address the reviewers’ concerns. We welcome any further questions or suggestions. After the rebuttal period, we will consolidate all feedback and make further revisions as needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.