acceptodds
Under review as a conference paper at ICLR 2027

Single-Configuration Evaluation Can Mislead Learned MILP Cut Selection: Seed Fragility, Measurement-Era Artifacts, and the Limits of Score Blending

Abstract

Learned components inside mixed-integer linear programming (MILP) solvers are often evaluated under a single experimental configuration, leaving the reliability of reported gains under solver and system variability unclear. We study this problem in the setting of learned cut selection in SCIP. Re-evaluating a frozen supervised cut scorer across six permutation seeds, we find substantial but sign-unstable effects on the search trajectory: only 2 of 31 test instances with complete six-seed data retain one benefit sign across all seeds, and pairwise sign agreement is 0.52. Two additional LightGBM cut scorers, trained with imitation and multiple-instance-learning targets, exhibit nearly identical sign instability (pairwise agreement 0.49 and 0.48, with no instance unanimous across all six seeds), ruling out our attribution-label recipe as the sole cause. Under a symmetric noise model, this disagreement places an approximately 60% upper bound on the fresh-seed sign accuracy of any seed-agnostic instance gate; the gates we test fail consistently with this ceiling. We uncover a second, orthogonal source of unreliability: an earlier 31/0 win–loss result against default SCIP was an artifact of a baseline measured under machine load. CPU-time accounting alone does not remove this confound, because memory pressure can inflate CPU cost per solver iteration; load logging and per-iteration rate probes expose it. After clean re-measurement, score blending is consistently less fragile than replacing SCIP's native score, but it does not produce robust downstream improvement: on benchmark-suite instances the intervention perturbs trajectories while about half of runs reach the same final primal and dual bounds as default, and on harder held-out instances the remaining positive structure is concentrated at intermediate difficulty and remains seed-dependent. These results motivate a two-axis evaluation protocol for learned solver components: vary solver symmetry-breaking conditions and verify the machine state under which paired comparisons are measured.

Primary area:
optimization
Keywords:
evaluation methodology, mixed-integer linear programming, cutting plane selection, SCIP, permutation seed, measurement era, seed robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6yKYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines the reliability of evaluating a learned cut selector in SCIP under a single experimental configuration. The authors start from their own earlier result. A gated LightGBM cut scorer had beaten default SCIP on 31/35 MIPLIB 2017 benchmark test instances with no losses at seed 42. They then re-evaluate this policy under six permutation seeds and inside a "clean", load-instrumented measurement window. They report four findings:

  1. The per-instance sign of the benefit is unstable across seeds: pairwise sign agreement is 0.52, and only 2/31 instances keep one sign across all seeds. The same pattern appears for two other LightGBM scorer recipes.
  2. Under a symmetric-noise model, Proposition 1 caps the fresh-seed sign accuracy of any seed-agnostic gate at about 60%.
  3. The original 31/0 result was an artifact of a baseline arm recorded under machine load (median 1.85× CPU-time inflation).
  4. A convex blend of the learned and native scores is less harmful than replacing the native score, but it gives no robust improvement.

The paper closes with a five-item evaluation checklist.

Critical strengths
  1. Transparency. Reporting a negative result, including the authors' own invalidated headline, is uncommon and valuable. The audit trail is documented carefully.
  2. Careful paired design in the final experiments. Configurations are frozen, results are reported per seed, the arms are paired by instance, seed and budget, and lower-tail statistics are given. The trajectory-versus-endpoint analysis (46–51% of pairs end with bounds identical to default) usefully separates "the search changed" from "the outcome improved".
  3. A cheap, practical diagnostic. The LP-iterations-per-CPU-second rate probe can be computed from existing logs and is easy for others to adopt.
  4. Candid limitations. The limitations section and the scoping of claims are honest.
Critical weaknesses
  1. The headline claim is not tested on the methods it concerns. The title and abstract make a claim about evaluating learned MILP cut selection in general. The evidence comes from a single LightGBM scorer (plus two close variants of the same family), applied only at the first root-separation round, to 35 test instances, on one workstation. None of the published learned cut selectors in Table 1 (Huang et al., 2022; Paulus et al., 2022; HEM; ACS) is re-evaluated under the proposed protocol. As it stands, the paper shows that one weak intervention has no robust effect. It does not show that single-configuration evaluation has misled the field. Re-evaluating even one published method with released code under the six-seed, load-verified protocol is the most important missing experiment.

  2. The sign instability is consistent with a near-null effect. In Table 3, SCIP's native hybrid score reaches a higher offline NDCG@30% (0.78) than all three learned scorers (0.72/0.67/0.53). This holds even against the attribution labels the main scorer was trained on. Add the near-zero clean-window means and the roughly half of runs that end with identical bounds, and the simplest explanation is that µ(i) ≈ 0 for most instances. In that case, pairwise sign agreement near 0.5 is exactly what is predicted by known MILP performance variability (Lodi & Tramontani, 2014). A placebo control is needed, measured under the same paired protocol. Examples: random reordering of the native top-k candidates, or default SCIP against default SCIP with a different randomseedshift. If the placebo also gives about 0.5, the finding is about null interventions rather than about learned components.

  3. Proposition 1 is elementary and its framing overstates what it shows. The result follows directly from A_i = p_i² + (1 − p_i)² and Jensen's inequality. More importantly, it bounds accuracy in predicting the realized sign on a fresh seed. That is not what a deployed gate needs. A gate should intervene where E[τ] = µ(i) > 0. A gate that knows sign(µ) exactly maximizes expected improvement even if its fresh-seed sign accuracy is only 60%. Also, µ(i) is identifiable by averaging over seeds; the obstacle is noise and sample size, not identifiability. The section title and the sentence "gating fails for lack of identifiability" should be revised. Finally, the seed-pair range of A is 0.29–0.74. Values well below 0.5 are hard to reconcile with the model, which implies A_i ≥ 0.5. Please discuss whether seeds are exchangeable and independent, as Proposition 1 assumes.

  4. Lemma 1 adds little. It is the standard minimum-variance combination of two correlated estimators. The paper does not argue why additive error variance after within-pool min-max normalization governs ranking quality, and it concedes that the lemma does not predict downstream benefit.

  5. The measurement-era finding is mainly an experimental-hygiene error. By the paper's own account, the oldest baseline runs predate CPU-time accounting, so queueing time entered the measurement. In addition, up to four concurrent solver runs were run on a 16 GB macOS workstation. Comparing arms recorded under different accounting and load conditions is invalid, which is why MIP benchmarking typically uses dedicated machines with exclusive runs. The rate probe is useful, but the main lesson is to avoid this setup in the first place. Two further points:

    • Please discuss deterministic effort measures, for example, integrating the primal–dual integral over LP iterations or nodes. These would sidestep the problem.
    • With four concurrent runs, even "clean" windows can suffer cache and memory-bandwidth contention. Confirming the key clean-window conclusions by running one job at a time on a subset would strengthen them.
  6. Statistical reporting is incomplete.

    • The abstract says blending is "consistently less fragile", but I could not find the blend's sign agreement or unanimity reported anywhere. Only a head-to-head win rate (0.46) is given. Please report the Section 8.2 sign metrics for α ∈ {0.25, 0.5, 0.75}.
    • Claims such as "neither statistically distinguishable from zero" should come with test statistics or confidence intervals (e.g., bootstrap over instances).
    • The 19 "active" collection instances are a post-hoc subset (both arms have finite PD). The result there, 11/19 positive, is consistent with chance and should be framed that way.
  7. The protocol contribution is incremental. The checklist restates established practice in MIP benchmarking (Koch et al., 2011; Lodi & Tramontani, 2014) and ML evaluation (Henderson et al., 2018; Agarwal et al., 2021). A quantitative contribution would make it more useful. For example: a power analysis of how many seeds and instances are needed to resolve an effect of a given size, using the variance components estimated here.

  8. Minor issues.

    • Clean-window rate ratios are given as 0.9×–1.0× in Section 8.3 but 0.85×–1.0× in Appendix A.
    • The training split is described as 163 instances, but the training set has rows from 114 instances (Section 6). Please clarify.
    • Much space goes to numbers that are later invalidated (31/0, +0.33/+0.31). These could move to an appendix.
    • The narrative framing ("incident", "autopsy", "era") makes it harder to separate contributions from project history.
Questions for authors
  1. Can you re-evaluate at least one published learned cut selector under your six-seed, load-verified protocol?
  2. What sign agreement does a placebo intervention achieve under the same protocol?
  3. What are the sign agreement and unanimity of the blend at α ∈ {0.25, 0.5, 0.75}?
  4. Given that the native hybrid score beats the learned scorer offline (NDCG 0.78 vs 0.72) on the learned scorer's own target, does this not already predict the online null result? Why was the learned scorer expected to help?
  5. Was the validation split, used to tune the gate thresholds, also re-measured in the clean window? Would the tuned thresholds change?
  6. Do the conclusions change if the primal–dual integral is computed over a deterministic effort measure instead of CPU time?
  7. Do the collection-split results hold when only one solver run executes at a time?
Reviewer_CgkCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors train a LightGBM cut scorer and plug it into SCIP's root-node cut selection. An initial single-seed evaluation showed a large win over default SCIP. The paper then reports what happened when they re-ran it. Across six permutation seeds, the sign of the per-instance benefit is close to a coin flip. The original win came from a baseline measured while the machine was under load. Blending the learned score with SCIP's native score at equal weight reduces the damage but does not improve on default SCIP. The paper adds a simple bound on how well an instance-level gate can predict the sign of the benefit (Proposition 1), and closes with a checklist for evaluating learned solver components.

Critical strengths
  1. Transparency. The authors report their own invalidated result, give per-seed outcomes rather than only aggregates, and release code. This is good practice, and the checklist in Section 9.1 is sensible.
Critical weaknesses
  1. The evidence does not support the title's claim. The paper argues that single-configuration evaluation can mislead learned cut selection. However, the only learned scorers it tests have no robust benefit once the baseline is re-measured. The paper itself finds "no regime with robust destination-level improvement" (l. 431).

    • In the clean window, the gated policy loses on its own training seed (Table 2), and at 600 s every blend weight has a negative mean effect relative to default (Fig. 3).
    • For an intervention with no real effect, a sign agreement close to 0.5 across seeds is exactly what one expects. It does not show that evaluation of a working method is fragile.
    • All three scorers are the authors' own LightGBM models (l. 467–469). No published cut selector is re-evaluated, and App. B states that the paper does not tabulate against published results (l. 633–634).
    • To support the claim, the paper would need to show sign instability for a component with a genuine average benefit.
  2. The contribution is limited.

    • Performance variability of MIP solvers under permutation seeds is well documented, as the paper itself notes (Lodi and Tramontani, 2014; Koch et al., 2011).
    • The "measurement-era artifact" is an experimental error: the baseline was run under memory pressure, and some of the oldest runs predate CPU-time accounting (l. 410). Running both arms under the same machine conditions is standard benchmarking practice, not a new finding.
    • Proposition 1 follows from Ai=pi2+(1−pi)2A_i = p_i^2 + (1-p_i)^2 and Jensen's inequality, and Lemma 1 is the variance of a convex combination of two noisy estimates.
  3. The manuscript does not read as an edited research paper. This is a major issue, not a cosmetic one.

    • The AI use statement says that generative tools drafted and edited parts of the manuscript and helped interpret the result tables (l. 488–491). The text reads as if this output was not substantially revised.
    • It is written as an incident report rather than a paper ("the incident", "the second autopsy", "measurement era", "churn without destination change"). The terminology is nonstandard and defined only in passing.
    • Invalidated numbers are mixed with valid ones throughout (e.g., Table 2 caption, §8.3, App. B "archived ablations" from "an earlier draft"). The reader has to keep track of which results still hold.
    • The abstract is very long and reads like a list of findings.
    • Figures 1–3 are too small to read, which suggests the PDF was not checked before submission.
Questions for authors
  1. Can you show the same sign instability for a cut-selection component that has a genuine positive average effect, for example a published learned selector or a SCIP setting known to help on these instances? Section 7 lists NoCuts and random, efficacy, and violation selectors among the evaluated policies (l. 293–294), but no results for them are reported. Please report at least the sign agreement of the random selector. If it is also close to 0.5, the learned scorers behave like a random perturbation of the search. Without these controls, the results are consistent with a null effect, and my assessment would not change.

  2. How many test instances have a mean effect across the six seeds that is significantly different from zero? If almost none, the instances satisfy μ(i)≈0\mu(i) \approx 0, which is outside the setting of Proposition 1 (μ(i)≠0\mu(i) \neq 0).

  3. Section 3 says the scorer was trained on 163 training instances, while Section 6 says the training data come from 114 instances. Which is correct?

Reviewer_kdcLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a negative-result methodology paper on learned cut selection in SCIP. (1). Re-running a frozen scorer under six permutation seeds yields near coin-flip per-instance benefit signs: pairwise sign agreement is 0.52, and only 2 of 31 instances keep one sign across all seeds. The same instability recurs under two other supervised recipes, so no single label design is to blame. (2). Under a symmetric noise model, Proposition 1 bounds the fresh-seed sign accuracy of any seed-agnostic instance gate at about 60%. (3). A second, orthogonal failure axis is measurement era: the original 31/0 win was an artifact of a baseline measured under machine load, with CPU times inflated by a median 1.85×, which CPU-time accounting alone cannot remove. After clean re-measurement, score blending proves less fragile than replacing the native score but delivers no robust downstream improvement.

Critical strengths
  1. The paper offers a cheap and clear way to test learned solver tools fairly: run several random seeds, report each seed's result, and check the machine's load — other researchers can easily reuse these checks .
  2. The theory matches the experiments: the predicted ~60% accuracy limit agrees with how the gates actually failed, and all three scorers showed the same instability, so the problem is not caused by one particular label design .
Critical weaknesses
  1. All tests were run on one solver (SCIP), one computer, and one model type (LightGBM), so the findings may not transfer to other solvers, machines, or models .
  2. The paper offers little that is constructive, and its contribution as a standalone submission is unclear. The paper is essentially a checklist plus a negative result. It gives no positive method that others can adopt, and no conclusion broad enough to change practice. To count as a methodology paper, it needs broader experiments across solvers, components, and tasks. To count as an ML4CO method paper, it needs a robust positive result. As it stands, it is neither.
Questions for authors
  1. If the intervention were made stronger/broader (more rounds, all nodes, no top-30% prefilter, or a state-conditioned policy), could robust downstream gains appear? How sensitive is the "no signal" conclusion to intervention strength?
  2. Do the conclusions reproduce on other solvers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.