- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
MILP-Evo uses LLM-driven evolutionary program search (Gemini 3 Flash) to write a PySCIPOpt module that implements a cut selector and a branching rule together.
- Search machinery. Islands, a MAP-Elites archive whose two descriptors are rated by the LLM, reflection prompts, mutation, and crossover.
- Validity. A "contract-preserving" repair operator fixes candidates that break PySCIPOpt callback contracts.
- Fitness. Mean SCIP process time over 10 Easy training instances, with invalid or unsolved candidates assigned infinite fitness.
- Evaluation. On four learn2branch families (set cover, combinatorial auctions, facility location, independent set), the exported module is tested on Easy, Medium and Hard scales: 10 instances × 5 seeds per scale. It has the best mean time in 7 of the 9 independent-set, auction, and set-cover settings.
- Analysis. Ablations on independent set compare joint search with single-component search and with search without repair. Repair statistics and a lineage analysis are also reported.
Critical strengths
- Inspectable, deployable output. The result is a readable callback module. The paper prints the full independent-set program and candidly analyzes which of its terms actually affect decisions (Appendix B.1).
- A sensible target for joint search. Searching for cut selection and branching together is well-motivated. On the independent set, the joint module clearly outperforms the post hoc combination of separately found components at the Hard scale (806 s vs 2,299 s).
- Appropriate statistics. Seeds are averaged per instance before exact Wilcoxon tests with Holm correction (Table 7), and the non-significant settings are reported openly.
- The repair operator is measured. Valid-candidate rate rises from 79.96% to 95.64% (Table 3).
Critical weaknesses
-
Methodological novelty over existing LLM program search is limited. Islands, MAP-Elites archives, reflection, and LLM mutation and crossover are established in FunSearch, EoH, ReEvo, and AlphaEvolve. The new parts are callback-contract checking and repair, which are engineering contributions. Please state what is methodologically new. Please also compare against at least one existing framework (for example, an EoH- or ReEvo-style loop) on the same callback space, with the same LLM and the same evaluation budget.
-
The most relevant baseline, algorithm configuration, is missing. Tuning solver parameters per problem family is a mature approach (SMAC, irace, ParamILS applied to SCIP or CPLEX). The evolved independent-set cut selector mainly limits how many cuts are added: it keeps cuts that score at least 85% of the best, capped at one-third of the pool. A similar effect may be reachable by tuning SCIP's separation and cut-selection parameters (maximum cuts per round, cut-selector weights, separation emphasis) together with branching parameters, under the same budget (400 iterations × 10 instances). In addition, every learned baseline modifies only branching, whereas MILP-Evo also changes cut selection, so the comparison conflates the two effects. A learned cut-selection baseline, or a branching-only MILP-Evo in the main table, is needed.
-
The baseline setup may not be consistent.
- The paper says all baselines use "the fixed defaults of their cited implementations", and the software versions it lists describe only "the main MILP-Evo environment". Please confirm that every method (FSB, RPB, GNN, Hybrid, GS4CO, SCIP) runs on the same SCIP 10.0.2 build, with the same parameter settings (for example, cuts at the root only and restarts disabled, as in learn2branch, versus SCIP defaults), on the same machines and thread counts.
- Some numbers suggest a mismatch. On an independent set at the Medium scale, GNN takes 1,673 s and Hybrid 1,502 s, compared to 194.74 s for RPB, roughly an 8× gap. The original learn2branch study found the GNN policy competitive with RPB at comparable scales.
- Please also explain how "RPB" and "SCIP" differ, since SCIP's default branching rule is reliability pseudo-cost branching.
-
The best result may rely on an instance-generator artifact. Appendix B.1 shows that the effective independent-set branching rule is (n_up(v), −i(v)): pick the variable with the most up-locks, then the smallest variable index. The learn2branch independent-set instances come from Barabási–Albert graphs. In that model, low-index nodes are the earliest added and tend to have the highest degree, so the index probably stands in for degree. That is a property of the generator's ordering, not of the problem. Please evaluate on instances whose variables and constraints are randomly permuted. If performance drops, the independent-set results, the paper's strongest, partly reflect ordering leakage. This also bears on the interpretability claim.
-
The evaluation is small and rests on one search run.
- Each scale uses only 10 test instances (× 5 seeds), fewer than the usual learn2branch protocol.
- Search uses 10 training instances and a single SCIP seed.
- Only one evolutionary run per family is reported. LLM-driven search is highly stochastic, so please report several independent runs (3–5) and the spread of their final test performance.
- Please report the total cost of search: input and output tokens (or dollars), plus CPU-hours of SCIP evaluation. Figure 2 shows about 600k output tokens for independent set, but the full cost is not given.
-
The ablations are under-specified. They are run only on independent set. It is unclear whether the cut-only and branch-only searches had the same 400-iteration budget, or whether NoRepair reached a comparable number of valid evaluations. Please clarify, and repeat the ablations on at least one more family.
-
Results are mixed, and the claims are broader than the evidence.
- MILP-Evo does not have the best mean time at the Hard scale of auctions (381.90 s vs 271.44 s for RPB) or set cover (2,641.34 s vs 2,401.92 s for RPB).
- It trails on facility location at the Hard scale (761.88 s vs 607.57 s).
- 5 of the 12 settings are not statistically significant.
- All evaluation is within a single synthetic family, transferring only from Easy to Medium and Hard; nothing is tested on heterogeneous benchmarks such as MIPLIB. The title, "Closed-Loop Fully Automatic Design of MILP Solvers", is therefore much broader than the evidence, which covers two callbacks, four synthetic families, and hand-written, family-specific prompts.
-
Presentation issues.
- The abstract gives no quantitative results.
- The formalization in Section 3.2 adds little.
- Table 1 is dense; mean ± std should be in the main text rather than only in Appendix C.1.
- The interpretability claim rests on one example, in which several terms turn out to be inactive.
- The overhead of Python callbacks (the share of time spent inside them) is not reported.
Questions for authors
- Do all baselines run on the same SCIP build, with the same parameter settings, on the same hardware? What explains GNN and Hybrid being so slow on independent set at the Medium scale?
- How does the independent-set module perform on instances with randomly permuted variables and constraints?
- How does MILP-Evo compare with SCIP parameter tuning (e.g., SMAC or irace) under a matched compute budget?
- What is the spread of results over several independent search runs, and what is the total search cost (tokens and CPU-hours)?
- Were the single-component and NoRepair ablations given the same budget as the full method?
- How does MILP-Evo compare with an existing LLM program-search framework (EoH, ReEvo, or FunSearch-style) on the same callback space?
- What fraction of solve time is spent inside the Python callbacks?
9 comments
3. Ordering sensitivity (Q2)
We tested the fixed exported independent-set policy without retraining or further LLM calls. Variable names, coefficients, bounds, objective, and variable types were preserved; only the input order was changed. Mathematical equivalence was verified after SCIP loaded each permuted instance.
For each scale, we used 10 instances, three permutation seeds (1101-1103), and five solver seeds (0-4), giving 150 runs per method with simultaneous variable/constraint permutations. MILP-Evo and SCIP used identical instances, permutations, and solver seeds, with one CPU core per run. We first averaged the three permutations within each instance/solver-seed pair, then calculated the permuted/original mean-time ratio across 50 matched pairs.
| Scale | MILP-Evo original (s) | MILP-Evo permuted (s) | Ratio | SCIP permuted (s) | Solved: MILP-Evo / SCIP |
|---|---|---|---|---|---|
| Easy | 3.05 | 3.23 | 1.056 | 8.88 | 150/150 / 150/150 |
| Medium | 68.46 | 62.75 | 0.917 | 244.16 | 150/150 / 150/150 |
| Hard | 879.03 | 869.40 | 0.989 | 2838.35 | 147/150 / 68/150 |
Times are CPU-PAR1, assigning 3,600 s to unsuccessful runs. The largest increase is 5.6% on Easy; Medium and Hard are 8.3% and 1.1% faster after permutation. On permuted Hard instances, MILP-Evo remains 3.26 times faster than SCIP and solves 147 rather than 68 runs. These results show that its advantage persists when the original input ordering is disrupted.
We also tested variable-only and constraint-only permutations on Easy instances with solver seed 0. Their permuted/original ratios are 1.094 and 1.028, respectively, providing separate checks of the two input orders.
4. Independent searches and cost (Q4)
Three 400-iteration searches yield best training fitness values (s) of 3.10/6.83/2.82 for independent set, 1.59/0.80/7.44 for auctions, 4.67/4.26/5.74 for set cover, and 25.43/23.31/24.72 for facility location.
Each search uses 3.14-3.75M recorded tokens and 8.90-36.87 evaluation CPU-hours, including invalid candidates; estimated elapsed time is 2.50-10.09 h.
The following ledger summarizes the three existing 400-iteration searches for each family. Token totals include input, output, and reasoning tokens; evaluation CPU time includes invalid candidates.
| Family / seed | Tokens (M) | Evaluation CPU (h) | Estimated elapsed time (h) |
|---|---|---|---|
| Independent set / 1001 | 3.23 | 10.51 | 3.15 |
| Independent set / 1002 | 3.31 | 12.43 | 3.71 |
| Independent set / 1003 | 3.14 | 8.92 | 2.56 |
| Auctions / 1001 | 3.42 | 11.02 | 3.02 |
| Auctions / 1002 | 3.75 | 8.90 | 2.50 |
| Auctions / 1003 | 3.62 | 14.93 | 4.08 |
| Set cover / 1001 | 3.37 | 24.88 | 6.93 |
| Set cover / 1002 | 3.41 | 18.91 | 4.95 |
| Set cover / 1003 | 3.44 | 30.16 | 7.78 |
| Facility location / 1001 | 3.36 | 36.82 | 9.68 |
| Facility location / 1002 | 3.69 | 36.22 | 10.09 |
| Facility location / 1003 | 3.60 | 36.87 | 9.77 |
CPU-hours measure accumulated solver-evaluation work; elapsed estimates describe the duration reported by the search ledger. These are offline costs. Once exported, the policy runs as solver callbacks without further LLM calls.
5. Ablation budgets (Q5)
Joint, Cut-only, Branch-only, and NoRepair each use 400 iterations. Separate-best is constructed by selecting the best individual from the Cut-only search and the best individual from the Branch-only search, then combining their cut-selection and branching components. Each component search uses 400 iterations, giving 800 iterations in total.
Table 2. Ablation mean process time (s), Easy/Medium/Hard; 50 runs per setting.
| Method | Independent set | Auctions | Set cover | Facility location |
|---|---|---|---|---|
| Joint | 2.57/59.20/806.17 | 0.81/11.79/381.90 | 3.14/89.36/2641.34 | 35.66/266.21/761.88 |
| Cut-only | 6.33/179.96/2522.53 | 3.50/26.63/259.96 | 10.44/185.67/2827.03 | 40.20/305.36/820.06 |
| Branch-only | 4.85/166.36/2452.77 | 3.04/26.40/239.26 | 10.18/205.81/2899.12 | 43.60/313.20/746.37 |
| Separate-best | 3.95/124.48/2299.23 | 2.40/16.52/227.42 | 8.80/192.48/2892.56 | 34.75/309.41/704.18 |
| NoRepair | 6.34/200.81/2615.15 | 3.07/28.14/248.14 | 9.37/215.65/2960.63 | 43.23/295.90/636.27 |
Joint leads nine settings, but Separate-best wins on Hard auctions and NoRepair on Hard facility location.
6. Statistics, overhead, and scope (Q7)
Across 120 Holm-corrected Wilcoxon comparisons using 20 instance means, MILP-Evo is significantly faster in 74, nonsignificant in 41, and slower in five. Losses are against COPT and GNN-GPU.
We profiled the fixed exported policies on five instances in each of the 12 family/scale settings, using solver seed 0: 60 runs in total, including 57 optimal completions and three timeouts. The numerator is cumulative wall time inside the cut-selection and branching callbacks, including their native API calls; nested callbacks are not double-counted. The denominator is wall time inside Model.optimize, excluding instance reading and model construction. Each setting-level percentage is the sum of callback times divided by the sum of optimize times.
| Family | Cut selection (%) E/M/H | Branching (%) E/M/H | Total (%) E/M/H |
|---|---|---|---|
| Independent set | 0.132/0.040/0.019 | 0.374/0.490/0.720 | 0.506/0.530/0.739 |
| Auctions | 0.115/0.046/0.031 | 1.353/1.272/1.550 | 1.467/1.318/1.581 |
| Set cover | 0.396/0.143/0.057 | 1.010/1.547/0.798 | 1.406/1.690/0.854 |
| Facility location | 0.027/0.004/0.001 | 0.034/0.016/0.004 | 0.061/0.019/0.005 |
Total callback time accounts for 0.005%-1.69% of optimize time across the 12 setting-level aggregates, showing that the deployed callbacks consume a small fraction of solving time. The table separates cut-selection and branching costs and includes native operations invoked inside the callbacks.
We will narrow the title and claims to family-specific co-design of two callbacks and within-family scale transfer. Cross-family/MIPLIB transfer remains untested. We will highlight Hard losses, add abstract numbers and main-table mean ± standard deviation, shorten Section 3.2, and qualify interpretability claims.
Thank you for the constructive review. We address the concerns below.
1. Contribution and search baselines (Q3, Q6)
Our contribution is joint synthesis of executable cut-selection and branching callbacks with solver-feedback validation and contract-preserving repair. The underlying LLM search operators are established.
ReEvo uses the same Gemini 3 Flash backbone and callback space as MILP-Evo, while SMAC tunes SCIP parameters. ReEvo is matched by total token budget and SMAC by solver-evaluation time, with each cap set to the corresponding MILP-Evo search budget. Tokens consumed by repair are included in the token budget.
Evaluation now covers 20 instances per scale and five runs per instance: 1,200 runs per method. Mean runtime includes every run, charging 3,600 s for unsuccessful runs.
Table 1. Aggregate results on the expanded benchmark.
MILP-Evo has lower mean runtime than ReEvo in all 12 settings and reduces aggregate mean runtime by 40.7% versus SMAC. COPT remains faster overall.
The following breakdown reports all 12 settings, each with 20 instances and five runs per instance. Unsuccessful runs contribute 3,600 s.
MILP-Evo improves mean runtime over ReEvo in every setting. The gains are particularly large on independent-set Hard (773.59 vs. 1767.91 s) and set-cover Medium (73.06 vs. 174.00 s). Against SMAC, MILP-Evo has lower mean runtime in 10 of 12 settings, while SMAC is faster on set-cover Easy and facility-location Medium. This breakdown shows where the aggregate 40.7% reduction over SMAC arises.
2. Baseline training and compatibility (Q1)
GNN/Hybrid followed official stopping criteria; independent-set training took 1h19m (67 epochs)/6h14m (61 epochs). GS4CO uses official pretrained expressions. All methods used the same hardware platform. Unrelated parameters were matched where possible; otherwise, version-specific defaults were retained.
MILP-Evo/ReEvo/SMAC/SCIP/FSB use SCIP 10.0.2, GNN 7.0.3, and Hybrid/GS4CO/RPB 6.0.1. RPB is relpscost in the GS4CO pipeline. MILP-Evo relies on the cut-selector plugin introduced in SCIP 8.0, so it cannot run unchanged on standard SCIP 6/7. Upgrading legacy baselines requires adapting PySCIPOpt/feature-extraction interfaces, including Ecole where used, and validating model/expression outputs and parameter mappings.
These constraints do not eliminate version effects. The same-version SCIP, ReEvo, and SMAC comparisons support our conclusions independently of the neural baselines. We will report parameter overrides and thread counts.
We additionally checked the independent-set Medium results on the new instances, pairing runs by instance and solver seed and retaining pairs solved to optimality by both methods.
The GNN CPU and GPU runs produce similar mean tree sizes, while GPU evaluation reduces runtime from 474.99 to 267.06 s. However, the GNN-GPU tree remains much larger than SCIP's on their jointly solved pairs. Hybrid also explores more nodes than RPB. Thus, the observed gap includes a search-tree difference and is not explained solely by neural inference overhead. Each row uses its own jointly solved subset.