- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates a real but somewhat niche issue in black-box subset optimisation, claiming that while existing methods can explore over choices of subset for fixed subset size, they struggle to explore beyond the subset sizes present in the initial data used to warm-start the optimisation.
The proposed solution is simple and just includes the size of the subset as an additional input with a weak prior. This is shown to work against on artificial test problems carefully constructed to have additive +ve structure. On the only more realistic benchmark, the method performs poorly.
Unfortunately, the manuscript is full of poor explanations and inconsistencies that make it hard to understand and thus assess.
Critical strengths
None
Critical weaknesses
-
The manuscript is very hard to follow. Key terms are used without definition or before they are defined. Much of the prose is compressed to the point of ambiguity (e.g., “a verdict on held-out instances,” “which sensor placement had no dispatch for”). The appendices read more like an experiment log than a structured supplement. A substantial rewrite is needed before the technical contribution can be properly assessed.
-
Limited novelty. The core observation is that an i.i.d. zero-mean prior on item coefficients assigns the common mean a prior variance of s²/n. The proposed fix, a hierarchical prior with a diffuse common mean, is textbook, as the citation to Gelman et al. acknowledges. The authors’ own GP experiment shows that simply adding a linear mean in |x| recovers about 100% of the size effect. Please clarify why this contribution is substantial enough for an ICLR paper?
-
The method’s additional components contribute little, meaning that C3BO effectively reduces to just adding an |x| feature with a diffuse prior and solving the resulting MIP.
-
I found several places where the manuscript contradicts itself or refers to content that is missing. These issues reduce my confidence in the reliability of results in the paper. (i)The introduction describes the four benchmarks as “standard constrained problems,” but Section 6.1 states they are generated by the authors rather than taken from a benchmark suite. (iii) The external-method set differs between experiments (Bounce appears in Table 4 but not Table 3; MALIBO and Meta-MO-TPE appear in Table 3 but not Table 4). (iv) The 3.2% figure in the abstract and the 6.4%/2.6% annotations in Figure 1 are not explained in the main text.
-
The paper repeatedly describes predictions and criteria as “registered” and uses this to support its credibility. However, what this means is unclear...
-
The benchmark problems are unrealistic. All four families use additive objectives with item values drawn uniformly from {10,…,99}, so every item effect shares one sign and the shared size effect is 2.1 standard deviations by construction. This is precisely the regime in which the proposed fix (and a linear surrogate) is advantaged, The evidence that C3BO helps, outside this synthetic regime is weak.
.
Questions for authors
The paper needs to be completely rewritten and tested on more realistic problems.
2 comments
Thank you for your response. I will maintain my score as I believe the advance to be incremental and I still think the paper requires a significant rewrite.
Response to Reviewer MCBW
Thank you for pointing out where the manuscript made the contribution and experimental protocol difficult to assess. We revised the presentation and added controlled comparisons for both the size prior and the fixed-size search.
W2. Technical contribution. The diffuse common-mean prior is standard. Our contribution is to identify a structural reason why common surrogates can miss good subset sizes and to test its effect on the decisions made during optimization. Proposition 1 shows that for binary-input kernels that depend only on which items differ, including RBF and Matérn kernels with ARD, the linear size direction receives exactly 1/n of the first-order prior variance, whenever that variance is nonzero. Tuning the ARD lengthscales leaves this fraction unchanged. Proposition 2 explains why a narrow range of observed sizes can leave the shared size effect strongly shrunk in a linear surrogate.
On the same stored archives, replacing only the fitted size effect with its true value increased the fraction of proposals within ten items of an optimal size from 0–9% to 63–91% (Table 9). In a separate online comparison without GP proposal replacement, removing the diffuse size prior reduced mean AUC by 14.2–35.7% with centring, intercept treatment, and search fixed (Table 20). The diagnosis also motivated an online GP experiment. Adding a linear size term to the qLogNEHVI GP mean improved mean AUC by 3.5–6.8% with the acquisition and candidate search fixed (Table 2).
W3. Contribution of fixed-size search. Disabling fixed-size search reduced mean hypervolume AUC by 13.77% across the five equally weighted OR-Library markets (nominal 95% market-bootstrap interval [−18.09%, −9.76%]). Repeat evaluations rose from 186 to 1,021, with 2,000 post-initial evaluations per variant. Both variants used the same prior, 2.5-second unrestricted solve limit, and duplicate handling. The ablated variant left the remaining solver time unused (Appendix H).
When the unrestricted MIP fails to certify an unseen design in time, C3BO uses the remaining budget to solve fixed-size MIPs and adds their candidates to the pool. A size-only error can change which size wins while leaving the ranking of designs within each fixed size unchanged (Algorithm 1, Lemma 4). Enabling the fixed-size solves improved mean AUC by 4.69% on Set Cover and 2.01% on MWIS, with smaller gains on the knapsacks (Table 1). On PyOED, the fallback was never called, and both variants evaluated the same designs in the same order (Appendix G).
W6. Evidence outside the original setting. On OR-Library, C3BO achieved higher mean AUC than the 14 external methods in Table 4 on all five markets. C3BO’s mean AUC across the five markets was 6.36% lower than that of a rule that samples near both feasible size limits (Appendix H). On PyOED, C3BO led in mean AUC on tomography and trailed several baselines on sensor placement. In the mixed-sign experiments at a mean-to-SD ratio of 0.5, fitting the size-prior scale improved mean AUC over centred L=0 by 4.8% on MOKP and 17.2% on Set Cover. At zero mean, the differences were −0.9% and +0.8%, respectively, with both intervals including zero at the reported precision. Both variants used the same GP proposal schedule. Table 18 reports the full comparison. These results clarify where the size-prior correction is supported.
W1, W4, and W5. Presentation and reporting. Algorithm 1 now shows the full batch procedure, and Section 6.1 defines the comparison policies. We identify the four main benchmarks as generated, clarify the baseline sets in Tables 1 and 4, and remove the unexplained Figure 1 annotations. Appendix K defines an optimal-size hit and reports the updated maximum of 44 out of 1,600 post-initial evaluations. Appendix J distinguishes comparisons prespecified in dated internal protocols from later amendments.
The submitted manuscript omitted the use of GP proposals in four of eight slots per batch for the final six batches of the four internal linear policies. We reran those policies without proposal replacement on the main and held-out generated instances (640 runs). Appendix E identifies which results use the reruns and which auxiliary controls retain the original schedule. The main conclusions on the generated benchmarks hold when each policy makes all its own proposals.