- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines the reliability of evaluating a learned cut selector in SCIP under a single experimental configuration. The authors start from their own earlier result. A gated LightGBM cut scorer had beaten default SCIP on 31/35 MIPLIB 2017 benchmark test instances with no losses at seed 42. They then re-evaluate this policy under six permutation seeds and inside a "clean", load-instrumented measurement window. They report four findings:
- The per-instance sign of the benefit is unstable across seeds: pairwise sign agreement is 0.52, and only 2/31 instances keep one sign across all seeds. The same pattern appears for two other LightGBM scorer recipes.
- Under a symmetric-noise model, Proposition 1 caps the fresh-seed sign accuracy of any seed-agnostic gate at about 60%.
- The original 31/0 result was an artifact of a baseline arm recorded under machine load (median 1.85× CPU-time inflation).
- A convex blend of the learned and native scores is less harmful than replacing the native score, but it gives no robust improvement.
The paper closes with a five-item evaluation checklist.
Critical strengths
- Transparency. Reporting a negative result, including the authors' own invalidated headline, is uncommon and valuable. The audit trail is documented carefully.
- Careful paired design in the final experiments. Configurations are frozen, results are reported per seed, the arms are paired by instance, seed and budget, and lower-tail statistics are given. The trajectory-versus-endpoint analysis (46–51% of pairs end with bounds identical to default) usefully separates "the search changed" from "the outcome improved".
- A cheap, practical diagnostic. The LP-iterations-per-CPU-second rate probe can be computed from existing logs and is easy for others to adopt.
- Candid limitations. The limitations section and the scoping of claims are honest.
Critical weaknesses
-
The headline claim is not tested on the methods it concerns. The title and abstract make a claim about evaluating learned MILP cut selection in general. The evidence comes from a single LightGBM scorer (plus two close variants of the same family), applied only at the first root-separation round, to 35 test instances, on one workstation. None of the published learned cut selectors in Table 1 (Huang et al., 2022; Paulus et al., 2022; HEM; ACS) is re-evaluated under the proposed protocol. As it stands, the paper shows that one weak intervention has no robust effect. It does not show that single-configuration evaluation has misled the field. Re-evaluating even one published method with released code under the six-seed, load-verified protocol is the most important missing experiment.
-
The sign instability is consistent with a near-null effect. In Table 3, SCIP's native hybrid score reaches a higher offline NDCG@30% (0.78) than all three learned scorers (0.72/0.67/0.53). This holds even against the attribution labels the main scorer was trained on. Add the near-zero clean-window means and the roughly half of runs that end with identical bounds, and the simplest explanation is that µ(i) ≈ 0 for most instances. In that case, pairwise sign agreement near 0.5 is exactly what is predicted by known MILP performance variability (Lodi & Tramontani, 2014). A placebo control is needed, measured under the same paired protocol. Examples: random reordering of the native top-k candidates, or default SCIP against default SCIP with a different
randomseedshift. If the placebo also gives about 0.5, the finding is about null interventions rather than about learned components. -
Proposition 1 is elementary and its framing overstates what it shows. The result follows directly from A_i = p_i² + (1 − p_i)² and Jensen's inequality. More importantly, it bounds accuracy in predicting the realized sign on a fresh seed. That is not what a deployed gate needs. A gate should intervene where E[τ] = µ(i) > 0. A gate that knows sign(µ) exactly maximizes expected improvement even if its fresh-seed sign accuracy is only 60%. Also, µ(i) is identifiable by averaging over seeds; the obstacle is noise and sample size, not identifiability. The section title and the sentence "gating fails for lack of identifiability" should be revised. Finally, the seed-pair range of A is 0.29–0.74. Values well below 0.5 are hard to reconcile with the model, which implies A_i ≥ 0.5. Please discuss whether seeds are exchangeable and independent, as Proposition 1 assumes.
-
Lemma 1 adds little. It is the standard minimum-variance combination of two correlated estimators. The paper does not argue why additive error variance after within-pool min-max normalization governs ranking quality, and it concedes that the lemma does not predict downstream benefit.
-
The measurement-era finding is mainly an experimental-hygiene error. By the paper's own account, the oldest baseline runs predate CPU-time accounting, so queueing time entered the measurement. In addition, up to four concurrent solver runs were run on a 16 GB macOS workstation. Comparing arms recorded under different accounting and load conditions is invalid, which is why MIP benchmarking typically uses dedicated machines with exclusive runs. The rate probe is useful, but the main lesson is to avoid this setup in the first place. Two further points:
- Please discuss deterministic effort measures, for example, integrating the primal–dual integral over LP iterations or nodes. These would sidestep the problem.
- With four concurrent runs, even "clean" windows can suffer cache and memory-bandwidth contention. Confirming the key clean-window conclusions by running one job at a time on a subset would strengthen them.
-
Statistical reporting is incomplete.
- The abstract says blending is "consistently less fragile", but I could not find the blend's sign agreement or unanimity reported anywhere. Only a head-to-head win rate (0.46) is given. Please report the Section 8.2 sign metrics for α ∈ {0.25, 0.5, 0.75}.
- Claims such as "neither statistically distinguishable from zero" should come with test statistics or confidence intervals (e.g., bootstrap over instances).
- The 19 "active" collection instances are a post-hoc subset (both arms have finite PD). The result there, 11/19 positive, is consistent with chance and should be framed that way.
-
The protocol contribution is incremental. The checklist restates established practice in MIP benchmarking (Koch et al., 2011; Lodi & Tramontani, 2014) and ML evaluation (Henderson et al., 2018; Agarwal et al., 2021). A quantitative contribution would make it more useful. For example: a power analysis of how many seeds and instances are needed to resolve an effect of a given size, using the variance components estimated here.
-
Minor issues.
- Clean-window rate ratios are given as 0.9×–1.0× in Section 8.3 but 0.85×–1.0× in Appendix A.
- The training split is described as 163 instances, but the training set has rows from 114 instances (Section 6). Please clarify.
- Much space goes to numbers that are later invalidated (31/0, +0.33/+0.31). These could move to an appendix.
- The narrative framing ("incident", "autopsy", "era") makes it harder to separate contributions from project history.
Questions for authors
- Can you re-evaluate at least one published learned cut selector under your six-seed, load-verified protocol?
- What sign agreement does a placebo intervention achieve under the same protocol?
- What are the sign agreement and unanimity of the blend at α ∈ {0.25, 0.5, 0.75}?
- Given that the native hybrid score beats the learned scorer offline (NDCG 0.78 vs 0.72) on the learned scorer's own target, does this not already predict the online null result? Why was the learned scorer expected to help?
- Was the validation split, used to tune the gate thresholds, also re-measured in the clean window? Would the tuned thresholds change?
- Do the conclusions change if the primal–dual integral is computed over a deterministic effort measure instead of CPU time?
- Do the collection-split results hold when only one solver run executes at a time?