acceptodds
Under review as a conference paper at ICLR 2027

When High Surrogate Scores Mislead: Pairwise Ranking Reliability for Offline Model-Based Optimization

Abstract

Many scientific and engineering design tasks require searching large design spaces for high-performing designs, yet costly evaluations leave only a fixed and sparsely distributed dataset. Offline model-based optimization (MBO) addresses this setting by learning surrogate models to guide candidate search. However, surrogate-guided optimization may produce misleading preferences when candidates lie beyond well-supported regions. Recent ranking-based methods improve candidate selection by learning relative candidate preferences, but a high surrogate score or rank does not indicate whether the induced pairwise ordering is reliable. Therefore, we propose TRACER, a two-stage framework for improving pairwise ranking reliability in offline MBO. Stage I improves surrogate order preservation through a Kolmogorov-Arnold Network (KAN) based surrogate, Top- contrastive regularization, and a two-stage candidate search strategy. We theoretically characterize the relationship between surrogate approximation error and pairwise ordering preservation, and further analyze how KAN's local spline structure supports order preservation in offline-supported regions. Stage II reassesses candidate comparisons after search by integrating reliability evidence from offline support and local prediction stability, and performs reliability-aware reranking with support-aware candidate refinement. Across five Design-Bench tasks, TRACER achieves the lowest mean rank among 18 baselines. The anonymous link of our source codes is available at https://anonymous.4open.science/status/TRACER-AF20.

Primary area:
optimization
Keywords:
Offline model-based optimization, Surrogate Model, Design-Bench

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oskiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that high surrogate scores in ranking-based offline MBO do not guarantee reliable pairwise orderings, even in well-supported regions. It proposes TRACER, a two-stage framework. Stage I improves order preservation with a KAN surrogate, Top-ρ contrastive regularization, and two-stage search. Stage II reranks candidates with a fuzzy-rule score combining predicted preference, offline support, and local stability, then moves weakly supported candidates toward the offline data. TRACER achieves the best mean rank on five Design-Bench tasks.

Critical strengths
  • Clearly identifies pairwise ranking reliability as an issue, with concrete reversal analysis across support levels.
  • Direct analysis in Section 4.2 clearly supports the need of two-stage algorithm.
  • Strong overall results, with the best mean rank against several recent baselines.
Critical weaknesses
  • Among several components, I think KAN contribution is marginal. Why should we use KAN-based surrogates?
  • Stage II is heuristic and heavy on hyperparameters such as α,β,f0\alpha, \beta, f_0. While I acknowledged that all of hyperparameters are fixed, I'm not sure that new benchmark can be well performed by these fixed hyperparameters.
Questions for authors
  • Please justify the reason why we should use KAN-based surrogates
  • Please justify the several hyperparameters and heuristic thresholds in stage 2.
2 comments
AuthorsOpenReview ↗

Responses to Reviewer oski (1/2)

Thank you for your constructive comments. We address your concerns below.

W1 & Q1: Why use a KAN surrogate, and how much does it contribute?

Response: Thank you for asking us to clarify the role of KAN and the size of its contribution. Stage I uses the surrogate to compare nearby designs, including designs that fall between observed samples. KAN offers a principled way to adjust the response to local supervision while strictly limiting where a spline-coefficient update can affect a given edge. We revised Section 3.1.2 to explain this property, added a case study on a trained surrogate, and report the size of the ranking gains in our controlled comparisons.

(1) Theoretical motivation for KAN. Theorem 1 in Section 3.1.2 characterizes the local influence of spline-coefficient updates on a single KAN edge. Each B-spline basis function is nonzero over a limited interval of the scalar edge input. With the edge inputs, knot grid, and branch weights fixed, updating the coefficients active at one input can therefore change the edge response only where at least one of those basis functions is nonzero. This gives the surrogate a principled way to adjust the shape of an edge response around inputs represented in the training data. The theorem concerns the direct effect of this coefficient update, and we evaluate the ranking quality of the trained surrogate empirically. The proof is in Appendix B.1.

(2) Case study on a KAN trained on Superconductor. Figure 2 shows how a change in individual coefficients affects the full edge response. We used 64 samples from Region A to take one gradient step on the spline coefficients of a trained edge, keeping the other parameters fixed. Only B4B_4 and B5B_5 were active on those samples, so only their coefficients changed. At the comparison input in Region B, both basis functions were zero. The full edge response therefore changed at the illustrated input in A and remained completely unchanged at B. Appendix D.3.7 reports the basis values and coefficient changes in Table 9. Figure 12 shows where the samples lie in a two-dimensional projection, explaining the difference in response.

(3) Size of the contribution under matched conditions. Figure 4(b) compares KAN and MLP surrogates trained on the same data with the same optimization settings and eight random seeds. Their trainable parameter counts differed by less than 0.7% (Appendix D.3.2). KAN achieved higher local pairwise ranking accuracy and Spearman correlation on both tasks.

TaskLocal pairwise accuracy, MLPLocal pairwise accuracy, KANSpearman, MLPSpearman, KAN
Ant0.9680.9840.9660.983
TF-Bind-80.9080.9150.9310.941

Figure 4(a) provides a complementary check on Superconductor. When the two surrogates ranked exactly the same candidate pairs, KAN reduced the reversal rate in each of the five offline-support groups by 0.2 to 0.7 percentage points. Although these reductions are modest, their consistency across all support groups, together with the capacity-matched results, strongly supports using KAN to improve local comparisons as a core component of Stage I.

AuthorsOpenReview ↗

Responses to Reviewer oski (2/2)

W2 & Q2: What justifies the Stage II hyperparameters and thresholds, and do the fixed settings transfer to a new benchmark?

Response: Thank you for raising this concern. We agree that keeping the settings fixed on Design-Bench alone gives limited evidence about their suitability on another benchmark. We clarified how the thresholds and reliability gate affect candidate selection, examined sensitivity to their numerical settings, and evaluated the original configuration on SOO-Bench without dataset-specific tuning. The theory explains which orderings the gate can change, while the sensitivity and transfer experiments provide strong evidence for the parameter choices.

(1) What the parameters control. Section 3.2 distinguishes two decisions. First, candidates that meet the thresholds for offline support FF, local prediction stability SS, and diagnostic priority TT receive precedence. Second, within each group, the weights α\alpha and β\beta combine surrogate preference QQ and diagnostic priority TT into a base priority UU. A gate GG, determined by support and stability, then scales UU. Theorem 2 gives the gate floor f0f_0 a concrete interpretation. For two candidates in the same group with Ui>Uj>0U_i > U_j > 0, candidate jj overtakes candidate ii exactly when Gj/Gi>Ui/UjG_j/G_i > U_i/U_j. At the default f0=0.25f_0=0.25, the largest possible gate ratio is four, so the gate cannot strictly reverse a fourfold or greater lead in base priority. Appendix B.2 gives the proof. Appendix C.3 and Tables 3 and 4 specify the fixed diagnostic rules and parameter settings.

Figure 3 illustrates why the two reliability signals enter the decision. On Superconductor, weak offline support lowers candidate A’s priority, while unstable local predictions lower candidate B’s priority even though it lies closer to the offline data. The signal ablation in Figure 6 evaluates this distinction on fixed candidate pools. Adding either signal reduced pairwise ranking errors on both tasks, and using both gave the lowest error rates.

(2) Evidence for the thresholds and pullback ratio. Figure 11(b) compares lower, default, and higher threshold settings on Superconductor. With the default (τF,τS,τT)=(0.55,0.60,0.28)(\tau_F, \tau_S, \tau_T) = (0.55, 0.60, 0.28), nScore100\mathrm{nScore}_{100} was 0.544, compared with 0.536 for the lower thresholds and 0.538 for the higher thresholds. This experiment evaluates the thresholds jointly. Figure 11(a) examines a separate decision made after selection. If a selected candidate has insufficient support or unstable predictions, the pullback ratio determines how far it moves toward its nearest offline reference. The default ratio of 0.35 yielded the highest nScore100\mathrm{nScore}_{100} (0.544) and nScore50\mathrm{nScore}_{50} (0.370) among the tested settings. Table 11 illustrates why repair can matter under a fixed budget. With 128 candidates for 128 slots, deleting flagged candidates left 109 submissions, while conditional pullback retained all 128.

(3) Sensitivity to the fusion weights and gate floor. Table 10 in Appendix D.4.3 reports our new sensitivity analysis on four GTOPX tasks. For each task and seed, all configurations used the same trained surrogate and candidate pool. We varied the fusion weights or gate floor while holding the thresholds and pullback ratio fixed. The entries below are percentage changes in mean SO\mathrm{SO} relative to the Design-Bench defaults.

Configuration (α,β,f0)(\alpha,\beta,f_0)GTOPX2GTOPX3GTOPX4GTOPX6
(0.4,0.6,0.25)(0.4, 0.6, 0.25), default0.00%0.00%0.00%0.00%
(0.2,0.8,0.25)(0.2, 0.8, 0.25)+0.03%−0.52%+0.30%−0.05%
(0.8,0.2,0.25)(0.8, 0.2, 0.25)−0.44%+0.32%−0.07%+0.14%
(0.4,0.6,0.75)(0.4, 0.6, 0.75)−0.79%+0.15%+0.23%−0.03%
(0,1,1)(0, 1, 1)−0.72%−1.52%−0.26%−0.75%

Across the three alternative non-boundary settings, the mean score stayed within 0.8% of the default on every task. The extreme setting (0,1,1)(0, 1, 1) assigns all base-priority weight to TT and sets the gate to one for every candidate. It reduced performance on all four tasks, with the largest decrease on GTOPX3. These results identify a tested range with little sensitivity and a boundary setting where performance deteriorates.

(4) Transfer to a new benchmark with the original settings. Table 2 in Section 4.2 directly addresses the concern about a new benchmark. We applied the Design-Bench hyperparameters to four unconstrained GTOPX tasks from SOO-Bench without additional tuning. We averaged TRACER and RaM-ListNet over eight independent seeds each and used the published results for the seven baselines from the original SOO-Bench study. On the benchmark’s SOω\mathrm{SO}_{\omega} metric, TRACER ranked first on GTOPX2, GTOPX4, and GTOPX6 and second on GTOPX3. Its mean rank of 1.25 among nine methods shows that the same configuration transferred successfully to these four tasks.

Reviewer_MwG7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Offline Model-based Optimization aims to find high-performing candidates for a black-box function using only an offline dataset. To address the offline MBO, this paper proposes a novel method, TRACER, to improve pairwise ranking reliability. TRACER includes 2 stages: Stage I trains a Kolmogorov-Arnold Network surrogate model using ListNet loss from RaM with contrastive regularization, and constructs a candidate pool. Stage II reranks these candidates using reliability evidence and refines candidates with weak support or unstable predictions. TRACER empirically achieves the lowest mean rank across 5 Design-Bench tasks.

Critical strengths
  • Training a Kolmogorov-Arnold Network surrogate model using a rank-based loss function is an interesting direction.
  • TRACER empirically achieves the best average rank across 5 Design-Bench tasks against 18 baselines.
Critical weaknesses
  • The contrastive regularization is somewhat questionable. The authors do not provide explicit intuition for why this regularization improves surrogate training.
  • The experiments are conducted only on the Design-bench benchmark. Additional results on other datasets will further prove the efficiency of TRACER.
Questions for authors
  1. What is BvB_v and VbV_b in Eq 4?

  2. The authors claim in lines 186–187 that “The perturbed candidates inherit the rank group of their original samples and are not assigned new objective values.” This seems questionable because a minor shift in a candidate might create a large difference in its value. For instance, in example B of Fig. 1, the distance between xqxq and xpxp is slight, yet the value gap is substantial. Consequently, adding Gaussian noise to a candidate could produce a major change in both its value and rank.

  3. What is the intuition of contrastive regularization?

  4. Other datasets and benchmarks are also used to evaluate the previous baseline in offline MBO, including RNA-Binding [1] or SOO-Bench [2]. Conducting additional experiments on these datasets will further prove the efficiency of TRACER.

[1] Ronny Lorenz, Stephan H. Bernhart, Christoph Höner Zu Siederdissen, Hanna Tafer, Christoph Flamm, Peter F. Stadler, and Ivo L. Hofacker. ViennaRNA Package 2.0. Algorithms for Molecular Biology, 6(1):26, November 2011.

[2] Qian, Hong, Yiyi Zhu, Xiang Shu, Shuo Liu, Yaolin Wen, Xin An, Huakang Lu, Aimin Zhou, Ke Tang, and Yang Yu. "SOO-bench: Benchmarks for evaluating the stability of offline black-box optimization." In International Conference on Learning Representations, vol. 2025, pp. 81720-81784. 2025.

2 comments
AuthorsOpenReview ↗

Responses to Reviewer MwG7 (1/2)

Thank you for your constructive comments. We address your questions below.

Q1: What do Bv\mathcal{B}_v and Vb\mathcal{V}_b denote in Eq. (4)?

Response: We apologize for the unclear notation. We have clarified this notation in Section 3.1. Furthermore, we have carefully checked all formulas and symbols throughout the manuscript to ensure consistency and clarity.

Q2: Why can a perturbed design inherit its original rank group if a small change can alter its objective value?

Response: Thank you for raising this critical concern. We agree that a small input change can substantially alter a design’s objective value. The assumption behind Top-ρ\rho concerns whether high-value designs generally continue to outrank designs outside the group under the small perturbations used for training. We clarified this assumption in Section 3.1.1 and tested it empirically.

(1) What the view inherits. The pair you highlight in Figure 1 illustrates this distinction well. The true values of xpx_p and xqx_q are 59.01 and 82.16, respectively. Assigning one design the other’s objective value based on proximity would indeed be unjustified. In TRACER, a perturbed view inherits only its source design’s group membership within the sampled list. It receives no objective value and no exact rank. For instance, two high-value designs can exchange positions and still provide consistent group supervision as long as both remain ahead of the designs outside the group. A reversal across the group boundary is the relevant failure mode.

(2) How often that assumption holds. Figure 10(a) in Appendix D.3.5 reports cross-group order preservation at eight Gaussian noise scales on Ant, Superconductor, and TF-Bind-8. Figure 10(b) reports the corresponding change in mean final optimization score, ΔnScore100\Delta\mathrm{nScore}_{100}, relative to zero noise. The default and largest tested scales illustrate the difference:

TaskOrder preserved, σv=0.01\sigma_v=0.01 (default)Order preserved, σv=0.30\sigma_v=0.30Score change, σv=0.01\sigma_v=0.01Score change, σv=0.30\sigma_v=0.30
Ant98.290%73.161%+0.0025−0.0323
Superconductor98.732%87.776%+0.0031+0.0040
TF-Bind-8100.000%99.740%0.0000−0.0045

At the default scale, all three tasks preserved more than 98% of cross-group orderings, and the largest absolute change in final score was 0.0031. At the largest scale, Ant lost more than a quarter of its cross-group orderings and its score dropped by 0.0323. These results strongly support using group inheritance at the default scale.

Q3: What is the intuition behind the contrastive regularizer?

Response: Thank you for asking us to clarify the motivation. Candidate search starts from promising offline designs and explores nearby regions where objective labels are unavailable. While the ranking loss (RankCosine) teaches the surrogate how to order the observed designs, Top-ρ\rho makes a second use of the same labels. It identifies which designs belong to the high-value group and uses this to supervise the representations of nearby unevaluated views.

(1) What supervision does Top-ρ\rho add? Consider two observed designs in the high-value group, where the first is slightly better than the second. RankCosine retains this fine-grained preference. For a perturbed view of the first design, the contrastive term encourages its representation to be similar to other high-value views (including views of the second design) and distinguishable from views outside the group. This provides a stable training signal for the unevaluated view without forcing it to inherit an exact, potentially incorrect, objective value.

(2) Does this improve local ranking? Yes. Table 7 in Appendix D.3.3 compares Top-ρ\rho with weight decay, dropout, consistency regularization, and InfoNCE. Top-ρ\rho achieved the highest mean local pairwise ranking accuracy and final optimization score on both tasks. For example, local pairwise ranking accuracy increased from 0.9519 with InfoNCE to 0.9807 with Top-ρ\rho on Ant, and from 0.9156 to 0.9254 on TF-Bind-8.

(3) What changes during candidate search? Figure 9 in Appendix D.3.4 compares search gradients learned with RankCosine alone versus with the added Top-ρ\rho term. The regularizer improved both directional and magnitude stability on all three evaluated tasks at all three tested perturbation scales. For example, on Ant at scale 0.05, directional stability increased from 0.724 to 0.783 and magnitude stability from 0.633 to 0.706 (Table 8). Thus, the ranking gains are accompanied by more consistent local guidance for search.

AuthorsOpenReview ↗

Responses to Reviewer MwG7 (2/2)

Q4: Can TRACER be evaluated on an additional benchmark beyond Design-Bench?

Response: Thank you for recommending an evaluation beyond Design-Bench. Following your suggestion, we added four unconstrained GTOPX tasks from SOO-Bench (Qian et al., 2025). We used the Design-Bench hyperparameters directly, without any dataset-specific tuning, and TRACER achieved the best mean rank on this additional benchmark.

(1) Evaluation protocol and baselines. We followed the original SOO-Bench protocol on GTOPX2, GTOPX3, GTOPX4, and GTOPX6. We evaluated TRACER and RaM-ListNet over eight independent random seeds each. For the other seven baselines, we used the results published in the original SOO-Bench paper. Table 2 therefore compares nine methods in total.

(2) Results with the Design-Bench hyperparameters. Using the benchmark’s SOω\mathrm{SO}_{\omega} metric, TRACER ranked first on GTOPX2, GTOPX4, and GTOPX6, and second on GTOPX3, achieving a mean rank of 1.25 among nine methods. The corresponding SOω\mathrm{SO}_{\omega} results for the two methods we evaluated are reproduced from Table 2 below:

MethodGTOPX2GTOPX3GTOPX4GTOPX6
RaM-ListNet1.33 ± 0.071.35 ± 0.021.34 ± 0.051.30 ± 0.05
TRACER1.47 ± 0.021.42 ± 0.041.45 ± 0.041.45 ± 0.05

The revised evaluation covers nine tasks across two benchmarks. The SOO-Bench results demonstrate that TRACER’s performance transfers effectively to new tasks under the same hyperparameter settings used on Design-Bench.

Reviewer_ZVhoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a two-stage approach to offline model-based optimization, aiming to improve the reliability of candidate rankings induced by learned surrogates. The first stage trains a KAN-based surrogate with a ranking loss, encourages local representation consistency via contrastive regularization, and generates initial candidates through a two-stage search procedure. The second stage heuristically combines surrogate preference, support distance, and local perturbation stability to rerank the candidates, select the most promising ones, and refine selected candidates by pulling them toward offline reference designs. The theoretical analysis relates surrogate approximation error to ranking reversals, provides support-dependent error bounds under explicit assumptions, and examines how KAN’s local spline structure may support order preservation. Experiments cover five Design-Bench tasks against 18 baselines, with the proposed method achieving a mean rank of 1.4/19.

Critical strengths
  1. Strong overall performance on Design-Bench tasks. The method demonstrates strong overall performance across continuous and discrete Design-Bench tasks. As reported in Table 1, it achieves the highest mean normalized score on four of the five tasks and a mean rank of 1.4 among 19 methods.

  2. Informative ablation studies. The authors provide informative ablation and controlled comparison studies including parameter-matched KAN/MLP comparisons and reranking comparisons on a shared candidate pool. These studies provide useful evidence for the contributions of the main components in both stages.

Critical weaknesses
  1. Limited novelty over prior work. The main ideas in both stages have clear precedents. Stage 1 builds on the ranking-based surrogate training framework in [1], while KAN-based surrogate learning with order-preserving training has already been explored in [2]. For Stage 2, reliability-based candidate selection and post-search design repair have been studied in [3] and [4], respectively. While TRACER combines these ideas differently, the two closely related Stage 2 methods are neither discussed nor compared, leaving its contribution over these methods unclear.

[1] Offline Model-Based Optimization by Learning to Rank. International Conference on Learning Representations (ICLR), 2025.

[2] Surrogate Learning in Meta-Black-Box Optimization: A Preliminary Study. Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), 2025.

[3] Beyond the Training Set: An Intuitive Method for Detecting Distribution Shift in Model-Based Optimization. arXiv preprint, 2023.

[4] Design Editing for Offline Model-based Optimization. Transactions on Machine Learning Research (TMLR), 2025.

  1. Insufficient validation of the Top-ρ\rho regularizer. The Top-ρ\rho regularizer places perturbed views in the same positive group as the original high-quality designs. However, in sensitive design spaces, small perturbations may substantially change objective values. This paper uses a fixed augmentation noise scale across tasks, but does not directly validate whether the perturbed designs remain suitable positive examples. Although sensitivity analyses of ρ\rho and the regularization weight are provided on Superconductor, sensitivity to the augmentation noise scale is not examined. This leaves the robustness of the regularizer across tasks with different local sensitivities unclear.
Questions for authors
  1. Could the authors examine sensitivity to the augmentation noise scale across multiple Design-Bench tasks?

  2. Stage 2 treats proximity to offline data and stability under local perturbations as evidence of reliability. However, stable predictions can still be wrong, while prediction variability may reflect objective sensitivity. Beyond the overall reranking gains, could the authors directly evaluate how well each signal helps identify pairwise ranking errors?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZVho (1/3)

Thank you, and we address your comments in Q&A:

[Reviewer 1 (ZVho), W1] Methodological novelty and distinction from prior work

Response: We sincerely thank the reviewer for highlighting these closely related studies. We would like to clarify TRACER’s contributions in two respects. First, we explain how the theoretical analysis supports the design of each stage. Second, we distinguish what each stage builds on from what it adds to existing methods and report the new comparison with DEMO. To make these contributions easier to follow, we reorganized the method and theoretical analysis in Section 3 around the two stages and revised the Abstract and Introduction accordingly. The changes are highlighted in blue in the revised manuscript.

1. Theoretical basis for the design of each stage

(1) Stage I. Where can a spline update change the surrogate’s response? We use a KAN surrogate so that training can adjust the response locally around promising designs. Theorem 1 in Section 3.1.2 characterizes this property on a single KAN edge. With the edge inputs, knot grid, and branch weights fixed, updating only the spline coefficients active at one input can change the edge response only where at least one of the corresponding basis functions is nonzero. Figure 2 illustrates this result using a KAN trained on Superconductor. An update using samples from Region A changed only the coefficients of B4B_4 and B5B_5. Both basis functions were zero at the comparison input in Region B, so the full edge response there remained unchanged. Appendix B.1 gives the proof. Appendix D.3.7 reports the coefficient changes in Table 9 and shows the sample locations in Figure 12.

(2) Stage II. How strong must reliability evidence be to reverse an existing priority? Theorem 2 in Section 3.2 establishes the exact reversal condition for two candidates in the same reliability group. Let Ui>Uj>0U_i>U_j>0 be their base priorities and Gi,GjG_i,G_j their gate values, determined by offline support and local prediction stability. Candidate jj overtakes candidate ii if and only if Gj/Gi>Ui/UjG_j/G_i>U_i/U_j. Thus, the larger the lead in base priority, the greater the relative gate advantage required to overturn it. Since the gate ratio is at most 1/f01/f_0, the default gate floor of f0=0.25f_0=0.25 prevents the gate from strictly reversing a fourfold or greater lead in base priority. Appendix B.2 gives the proof. Figure 5(a) in Section 4.2 separately evaluates whether reranking improves agreement with the true objective values. Stage II corrected the order of 17.39% of all evaluated pairs and introduced errors on 2.81%.

2. Methodological differences from the four cited studies and the new empirical comparison

(1) Stage I and prior ranking-based surrogates [1, 2]. As Figure 1(c) illustrates, RaM can rank the inferior of two candidates first even when both lie close to previously evaluated designs. We retain RaM’s RankCosine objective [1] to learn the order of observed designs. Section 3.1.1 explains what TRACER adds. It uses the observed labels to identify a high-value group within each sampled list and extends that group information to nearby unevaluated views without assigning them objective values. Surr-RLDE [2] already trains a KAN surrogate with an order-aware loss. Our contribution concerning KAN is the analysis of local coefficient updates described above, together with a capacity-matched comparison of ranking quality. Figure 4(b) shows that KAN achieved higher local pairwise ranking accuracy and Spearman correlation than the MLP on both Ant and TF-Bind-8.

(2) Stage II and prior post-search methods [3, 4]. Damani et al. [3] filter candidates by distribution shift and select the highest-scoring designs that remain. DEMO [4] uses a learned diffusion prior to edit designs affected by overoptimization. Section 3.2 explains how TRACER uses offline support and local prediction stability to reassess the order of candidates competing for evaluation slots. After selection, it moves a candidate toward its nearest offline reference only if its support or stability falls below the corresponding threshold. The candidate retains its evaluation slot. We discuss both studies in the revised Introduction and added a direct comparison with DEMO using the authors’ publicly available implementation. Table 1 reports higher mean normalized scores for TRACER on all five Design-Bench tasks, including 0.544 versus 0.501 on Superconductor and 0.991 versus 0.950 on TF-Bind-8. Among 20 methods, the mean ranks were 1.40 for TRACER and 8.20 for DEMO.

AuthorsOpenReview ↗

Response to Reviewer ZVho (2/3)

W2 & Q1: Validity of Top-ρ\rho group supervision and sensitivity to augmentation noise

Response: Thank you for raising the concern that a small perturbation may change a design’s value enough to invalidate its high-value group assignment. We agree that the usefulness of this supervision depends on how sensitive the task is to local changes. Following your suggestion, we clarified what a perturbed view inherits and evaluated both the reliability of that supervision and its effect on final optimization across tasks. At the default small noise scale, cross-group orderings were largely preserved and final scores changed little. Larger perturbations could disrupt those orderings and impair optimization.

(1) What a perturbed view inherits. Section 3.1.1 clarifies that observed objective values identify the high-value group within each sampled list. RankCosine supervises the fine-grained ordering of the observed designs. Top-ρ\rho uses the group identities to bring high-value views closer in representation space and distinguish them from the remaining views. A perturbed view receives no objective label. For example, two high-value designs may exchange positions after perturbation while both still outrank designs outside the group. Their group identities remain useful in this case. The relevant test is how often a perturbation reverses an ordering across the group boundary.

(2) Do cross-group orderings survive perturbation? Figure 10(a) in Appendix D.3.5 reports cross-group order preservation at eight Gaussian noise scales on Ant, Superconductor, and TF-Bind-8. At the default scale of σv=0.01\sigma_v=0.01 in normalized input coordinates, preservation was 98.290%, 98.732%, and 100.000%, respectively. At σv=0.30\sigma_v=0.30, it fell to 73.161% on Ant and 87.776% on Superconductor, while remaining at 99.740% on TF-Bind-8. These results support the group assignment at the default scale and show where the reviewer’s concern becomes consequential.

(3) How does noise affect the final optimization score? Figure 10(b) reports the change in mean nScore100\mathrm{nScore}_{100} relative to zero noise. The zero-noise setting retains the Top-ρ\rho loss, allowing us to isolate the effect of perturbing the training views. At the default scale, the largest absolute change across the three tasks was 0.0031. At the largest scale, Ant’s score fell by 0.0323, whereas Superconductor’s remained 0.0040 above its zero-noise reference. Thus, stronger noise can substantially disrupt group supervision, while its effect on the final designs varies by task.

Training noise σv\sigma_vAntSuperconductorTF-Bind-8
00.00000.00000.0000
0.005+0.00070.00000.0000
0.01 (default)+0.0025+0.00310.0000
0.03+0.0052−0.00040.0000
0.05+0.0061+0.00200.0000
0.10−0.0031+0.01200.0000
0.20−0.0160−0.00200.0000
0.30−0.0323+0.0040−0.0045

(4) Does group supervision improve ranking quality? Table 7 in Appendix D.3.3 compares Top-ρ\rho with weight decay, dropout, consistency regularization, and InfoNCE. Top-ρ\rho achieved the highest mean local pairwise ranking accuracy and final optimization score on both evaluated tasks. For example, compared with InfoNCE, local pairwise ranking accuracy increased from 0.9519 to 0.9807 on Ant and from 0.9156 to 0.9254 on TF-Bind-8.

AuthorsOpenReview ↗

Response to Reviewer ZVho (3/3)

Q2: How informative are offline support and local prediction stability about pairwise ranking errors?

Response: Thank you for pointing out that stable predictions can still be wrong and that variation under perturbation may reflect genuine sensitivity of the objective. We agree that either signal alone gives an incomplete picture of whether two candidates are correctly ordered. To address your question, we evaluated the contribution of each signal to reranking and checked the resulting orderings against the benchmark objective values. Both signals helped correct ranking errors, and their combination gave the most accurate ordering on both tasks.

(1) Individual and combined contributions of the two signals. Figure 6(a) in Section 4.2 compares four configurations on the same candidate pool for each task. Starting from surrogate preference alone (QQ), we add offline support (Q+FQ+F), local prediction stability (Q+SQ+S), or both (Q+F+SQ+F+S). Across Ant and Superconductor, adding FF reduced the pairwise error rate by 9.94 to 10.00 percentage points. Adding SS reduced it by 8.98 to 9.01 points. The reduction reached 14.60 to 15.00 points when both were used. Figure 6(b) shows that these changes also benefited the designs selected for evaluation. With all three signals, the mean optimization score increased from 0.946 to 0.972 on Ant and from 0.523 to 0.535 on Superconductor.

(2) Why the two signals are complementary. Figure 3 follows three actual candidates from Superconductor. Candidate A ranks first by surrogate score but falls to 128th when its weak offline support is taken into account. Candidate B still ranks fourth at that point. Its predictions fluctuate under local perturbations, and it falls to 69th when stability is included. Candidate C has both strong offline support and stable predictions and rises from eighth in the original ranking to first. The case illustrates why proximity can prompt us to reconsider A, while prediction stability supplies additional evidence for reconsidering B.

(3) How often does reranking correct an error or introduce one? Figure 5(a) tracks both outcomes on the same candidate pairs. Stage II changed an incorrect ordering to a correct one on 17.39% of all evaluated pairs and changed a correct ordering to an incorrect one on 2.81%. The net reduction was 14.58 percentage points. These results quantify the value of support and stability as evidence for a reranking decision. The accuracy of either signal as a stand-alone error detector remains a separate question.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Responses & Additional Experiments (1/2)

We sincerely thank the reviewers for their constructive comments and for recognizing the value of our work. We are particularly grateful to reviewer ZVho for noting that our method “demonstrates strong overall performance” and for acknowledging that we provided “informative ablation studies.” We also thank reviewer MwG7 for pointing out that our work is an “interesting research direction” and for confirming “empirically achieves the best average rank”. Furthermore, we thank reviewer oski for noting that we “Clearly identifies pairwise ranking reliability as an issue” and that “Direct analysis in Section 4.2 clearly supports the need of two-stage algorithm.”

Based on the reviewers’ suggestions and questions, we made the following revisions and conducted additional experiments. We have incorporated these changes into the manuscript, highlighted them in blue, and uploaded the revised version to the submission system. A link to our anonymized code is provided in the abstract, and all data and code will be made publicly available with the final manuscript.

  • [Reviewer 1 (ZVho), W1] Motivation, contributions, and comparison with prior work. (1) We revised the Abstract and Introduction to clarify TRACER’s contributions. The Introduction organizes prior work into three lines and explains why pairwise ranking reliability remains an issue even when both candidates have strong offline support (Fig. 1(c)). We also clarify what Stage I adds to prior ranking-based surrogate methods, including RaM and Surr-RLDE. (2) We reorganized the method and theoretical analysis in Section 3 around the objectives of the two stages to clarify how each stage contributes to pairwise ranking reliability and how the theoretical results support its design. (3) We discussed both post-search methods suggested by the reviewer. To ensure a fair and rigorous empirical evaluation, we added a direct comparison with DEMO using the authors’ publicly available implementation, where TRACER achieved higher mean scores than DEMO on all five Design-Bench tasks (Table 1). For the distribution-shift filtering method proposed by Damani et al. [3], since their official source code is not publicly available, we focused on a detailed methodological discussion rather than an unverified reimplementation. Specifically, we clarify that while Damani et al. rely on a single distribution-shift score to filter out candidates, our Stage II integrates multiple reliability signals (offline support and local prediction stability) into a bounded multiplicative gate, which provides a more comprehensive and robust assessment of pairwise ranking reliability.

  • [Reviewer 1 (ZVho), W2 and Q1] [Reviewer 2 (MwG7), W1 and Q1–Q3] Top-ρ\rho regularization and sensitivity to augmentation noise. (1) We clarified the notation in Eq. (4) and the motivation for Top-ρ\rho. (2) We compared Top-ρ\rho with four alternative regularizers. It achieved the highest local pairwise ranking accuracy and final optimization score on both evaluated tasks (Table 7). (3) We tested eight noise scales across three tasks. At the default small noise scale, cross-group orderings were largely preserved and final scores changed little (Fig. 10).

  • [Reviewer 1 (ZVho), Q2] [Reviewer 3 (oski), W2 and Q2] Stage II reliability signals and parameter settings. (1) We clarified the theoretical basis (i.e., Theorem 2) for the Stage II in Section 3.2. Our Stage II reranking algorithm is built upon this bounded multiplicative gate (Finali=UiGi\mathrm{Final}_i = U_i G_i) rather than a weighted sum, which avoids arbitrary weights and protects high-value candidates from being over-penalized by reliability scores. Theorem 2 shows that reversing a ranking requires the lower-ranked candidate’s gate advantage (Gj/GiG_j/G_i) to outweigh the base-priority gap (Ui/UjU_i/U_j). Since the maximum possible gate ratio is 1/f01/f_0, a candidate with a fourfold or greater lead in base priority (Ui/Uj≥4U_i/U_j \ge 4 for f0=0.25f_0=0.25) cannot reverse this ordering, no matter how strong the reliability evidence is. (2) Figure 3 illustrates how offline support and local prediction stability affect selection using actual candidates from Superconductor. Weak offline support lowers A’s rank, while unstable local predictions lower B’s rank, bringing C to first place. (3) Figure 6 presents our new ablation study on Ant and Superconductor. Adding either signal to surrogate preference reduced pairwise ranking errors and improved the final optimization score on both tasks. Using both signals yielded the lowest error rates and the highest scores. These results complement the fixed-pool comparison in Figure 5(a), where reranking corrected the order of 17.39% of all candidate pairs and introduced errors in 2.81%. (4) Table 10 reports the new sensitivity analysis of the fusion weights. Figures 11(a) and 11(b) examine sensitivity to the pullback ratio and reliability thresholds, respectively, on Superconductor.

AuthorsOpenReview ↗

Summary of Responses & Additional Experiments (2/2)

  • [Reviewer 3 (oski), W1 and Q1] Theoretical and empirical justification for the KAN surrogate. (1) We clarified the motivation for using KAN through Theorem 1 in Section 3.1.2. Theorem 1 explains why spline updates on a single KAN edge can have a local effect. With the knot grid and branch weights fixed, an update to the coefficients of the basis functions active at vv can change the edge response only where at least one of those functions is nonzero. (2) We added a case study on a KAN trained on Superconductor. Figure 2 shows which basis functions are active, which coefficients change, and how their combined contribution changes the full edge response at one input while leaving it unchanged at the other. Appendix D.3.7 reports the coefficient changes (Table 9) and shows the locations of the samples used in the comparison (Fig. 12). (3) Figure 4(b) shows that, under matched parameter budgets, the KAN surrogate achieved higher local pairwise ranking accuracy and Spearman correlation than the MLP surrogate on both Ant and TF-Bind-8.

  • [Reviewer 2 (MwG7), W2 and Q4] [Reviewer 3 (oski), W2] Evaluation on an additional benchmark. (1) We evaluated TRACER on four unconstrained GTOPX tasks from SOO-Bench using the same hyperparameters as on Design-Bench. Table 2 reports results averaged over eight random seeds for TRACER and RaM-ListNet, alongside the published results for the seven baselines evaluated in the original SOO-Bench paper. (2) TRACER ranked first on three tasks and second on the fourth, achieving the best mean rank of 1.25 among all nine methods. This result shows that TRACER’s ranking advantage extends to SOO-Bench without dataset-specific tuning, supporting its generalization across different offline data distributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.