- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that high surrogate scores in ranking-based offline MBO do not guarantee reliable pairwise orderings, even in well-supported regions. It proposes TRACER, a two-stage framework. Stage I improves order preservation with a KAN surrogate, Top-ρ contrastive regularization, and two-stage search. Stage II reranks candidates with a fuzzy-rule score combining predicted preference, offline support, and local stability, then moves weakly supported candidates toward the offline data. TRACER achieves the best mean rank on five Design-Bench tasks.
Critical strengths
- Clearly identifies pairwise ranking reliability as an issue, with concrete reversal analysis across support levels.
- Direct analysis in Section 4.2 clearly supports the need of two-stage algorithm.
- Strong overall results, with the best mean rank against several recent baselines.
Critical weaknesses
- Among several components, I think KAN contribution is marginal. Why should we use KAN-based surrogates?
- Stage II is heuristic and heavy on hyperparameters such as . While I acknowledged that all of hyperparameters are fixed, I'm not sure that new benchmark can be well performed by these fixed hyperparameters.
Questions for authors
- Please justify the reason why we should use KAN-based surrogates
- Please justify the several hyperparameters and heuristic thresholds in stage 2.
2 comments
Responses to Reviewer oski (2/2)
W2 & Q2: What justifies the Stage II hyperparameters and thresholds, and do the fixed settings transfer to a new benchmark?
Response: Thank you for raising this concern. We agree that keeping the settings fixed on Design-Bench alone gives limited evidence about their suitability on another benchmark. We clarified how the thresholds and reliability gate affect candidate selection, examined sensitivity to their numerical settings, and evaluated the original configuration on SOO-Bench without dataset-specific tuning. The theory explains which orderings the gate can change, while the sensitivity and transfer experiments provide strong evidence for the parameter choices.
(1) What the parameters control. Section 3.2 distinguishes two decisions. First, candidates that meet the thresholds for offline support , local prediction stability , and diagnostic priority receive precedence. Second, within each group, the weights and combine surrogate preference and diagnostic priority into a base priority . A gate , determined by support and stability, then scales . Theorem 2 gives the gate floor a concrete interpretation. For two candidates in the same group with , candidate overtakes candidate exactly when . At the default , the largest possible gate ratio is four, so the gate cannot strictly reverse a fourfold or greater lead in base priority. Appendix B.2 gives the proof. Appendix C.3 and Tables 3 and 4 specify the fixed diagnostic rules and parameter settings.
Figure 3 illustrates why the two reliability signals enter the decision. On Superconductor, weak offline support lowers candidate A’s priority, while unstable local predictions lower candidate B’s priority even though it lies closer to the offline data. The signal ablation in Figure 6 evaluates this distinction on fixed candidate pools. Adding either signal reduced pairwise ranking errors on both tasks, and using both gave the lowest error rates.
(2) Evidence for the thresholds and pullback ratio. Figure 11(b) compares lower, default, and higher threshold settings on Superconductor. With the default , was 0.544, compared with 0.536 for the lower thresholds and 0.538 for the higher thresholds. This experiment evaluates the thresholds jointly. Figure 11(a) examines a separate decision made after selection. If a selected candidate has insufficient support or unstable predictions, the pullback ratio determines how far it moves toward its nearest offline reference. The default ratio of 0.35 yielded the highest (0.544) and (0.370) among the tested settings. Table 11 illustrates why repair can matter under a fixed budget. With 128 candidates for 128 slots, deleting flagged candidates left 109 submissions, while conditional pullback retained all 128.
(3) Sensitivity to the fusion weights and gate floor. Table 10 in Appendix D.4.3 reports our new sensitivity analysis on four GTOPX tasks. For each task and seed, all configurations used the same trained surrogate and candidate pool. We varied the fusion weights or gate floor while holding the thresholds and pullback ratio fixed. The entries below are percentage changes in mean relative to the Design-Bench defaults.
| Configuration | GTOPX2 | GTOPX3 | GTOPX4 | GTOPX6 |
|---|---|---|---|---|
| , default | 0.00% | 0.00% | 0.00% | 0.00% |
| +0.03% | −0.52% | +0.30% | −0.05% | |
| −0.44% | +0.32% | −0.07% | +0.14% | |
| −0.79% | +0.15% | +0.23% | −0.03% | |
| −0.72% | −1.52% | −0.26% | −0.75% |
Across the three alternative non-boundary settings, the mean score stayed within 0.8% of the default on every task. The extreme setting assigns all base-priority weight to and sets the gate to one for every candidate. It reduced performance on all four tasks, with the largest decrease on GTOPX3. These results identify a tested range with little sensitivity and a boundary setting where performance deteriorates.
(4) Transfer to a new benchmark with the original settings. Table 2 in Section 4.2 directly addresses the concern about a new benchmark. We applied the Design-Bench hyperparameters to four unconstrained GTOPX tasks from SOO-Bench without additional tuning. We averaged TRACER and RaM-ListNet over eight independent seeds each and used the published results for the seven baselines from the original SOO-Bench study. On the benchmark’s metric, TRACER ranked first on GTOPX2, GTOPX4, and GTOPX6 and second on GTOPX3. Its mean rank of 1.25 among nine methods shows that the same configuration transferred successfully to these four tasks.
Responses to Reviewer oski (1/2)
Thank you for your constructive comments. We address your concerns below.
W1 & Q1: Why use a KAN surrogate, and how much does it contribute?
Response: Thank you for asking us to clarify the role of KAN and the size of its contribution. Stage I uses the surrogate to compare nearby designs, including designs that fall between observed samples. KAN offers a principled way to adjust the response to local supervision while strictly limiting where a spline-coefficient update can affect a given edge. We revised Section 3.1.2 to explain this property, added a case study on a trained surrogate, and report the size of the ranking gains in our controlled comparisons.
(1) Theoretical motivation for KAN. Theorem 1 in Section 3.1.2 characterizes the local influence of spline-coefficient updates on a single KAN edge. Each B-spline basis function is nonzero over a limited interval of the scalar edge input. With the edge inputs, knot grid, and branch weights fixed, updating the coefficients active at one input can therefore change the edge response only where at least one of those basis functions is nonzero. This gives the surrogate a principled way to adjust the shape of an edge response around inputs represented in the training data. The theorem concerns the direct effect of this coefficient update, and we evaluate the ranking quality of the trained surrogate empirically. The proof is in Appendix B.1.
(2) Case study on a KAN trained on Superconductor. Figure 2 shows how a change in individual coefficients affects the full edge response. We used 64 samples from Region A to take one gradient step on the spline coefficients of a trained edge, keeping the other parameters fixed. Only B4 and B5 were active on those samples, so only their coefficients changed. At the comparison input in Region B, both basis functions were zero. The full edge response therefore changed at the illustrated input in A and remained completely unchanged at B. Appendix D.3.7 reports the basis values and coefficient changes in Table 9. Figure 12 shows where the samples lie in a two-dimensional projection, explaining the difference in response.
(3) Size of the contribution under matched conditions. Figure 4(b) compares KAN and MLP surrogates trained on the same data with the same optimization settings and eight random seeds. Their trainable parameter counts differed by less than 0.7% (Appendix D.3.2). KAN achieved higher local pairwise ranking accuracy and Spearman correlation on both tasks.
Figure 4(a) provides a complementary check on Superconductor. When the two surrogates ranked exactly the same candidate pairs, KAN reduced the reversal rate in each of the five offline-support groups by 0.2 to 0.7 percentage points. Although these reductions are modest, their consistency across all support groups, together with the capacity-matched results, strongly supports using KAN to improve local comparisons as a core component of Stage I.