- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies self-supervised MILP representation learning using equivalent reformulations. It characterizes universally valid affine augmentation rules and separates constraint re-descriptions from variable substitutions. ReMILP combines a graph encoder with a hypernetwork that predicts changes in variable embeddings under substitutions. Experiments evaluate reformulation retrieval and transfer to binary solution, constraint activity, and integrality-gap prediction, using both frozen and fine-tuned encoders.
Critical strengths
The distinction between re-descriptions and substitutions gives the learning objective a concrete motivation. Making all equivalent formulations invariant would ignore the fact that a change of variables also changes what each variable represents. The characterization in Section 4.1 is useful in its own right, and the paper is explicit about its restriction to affine rules that work for every instance.
There is evidence that the representations contain useful information beyond what a random encoder provides. The frozen integrality-gap result is particularly encouraging: MAE falls from 0.063 for Random and 0.053 for FORGE to 0.040 for ReMILP. The shared backbone and the additional FORGE pooling comparison help make this result interpretable. Appendix B also provides a useful comparison against pre-training with redundant constraints alone.
Critical weaknesses
The experiments do not yet isolate the value of learning an equivariant action. Appendix B removes substitutions, the hypernetwork, and the action negatives together. It establishes a benefit from the full substitution-based setup over re-description-only pre-training, but does not establish whether a standard invariant contrastive objective on the same substituted views would perform worse. This matters because the invariance/equivariance distinction is the main methodological argument. Table 1 shows improved retrieval with the hypernetwork, but that is a closely related pre-training diagnostic, and its Top-1 accuracy is still 0.162. It does not resolve whether learning the action is what improves downstream prediction. A comparison using identical reformulated views but an invariant objective would be more decisive.
The scope of the transfer evidence needs more care. Section D.6 selects pre-training hyperparameters through 56 trials per method, using labeled downstream validation performance on seven D-MIPLIB families. Thus, although the encoder loss uses no solver labels, the complete selection procedure does. Table 1 then aggregates retrieval on all 23 families, including those used for selection, under the heading of unseen classes. This is not test-set leakage, but it is weaker evidence of class-level generalization than the wording suggests. MILP-Evolve is also split randomly by instance (Section C.1), rather than by family. Results separated into selection families and untouched problem classes would help establish how broadly the learned prior transfers.
Questions for authors
- How does an invariant contrastive model perform when trained on exactly the same substituted and re-described views as ReMILP? Appendix B changes the augmentation distribution as well as the objective, so it cannot answer this question.
- Could you report retrieval and downstream results separately for problem classes used in hyperparameter selection and classes excluded from selection? For MILP-Evolve, how much instance-family overlap is there across the pre-training and evaluation splits?
3 comments
Hyperparameter Tuning and Transfer to Unseen Classes
W2 & Q2: Hyperparameter Tuning and Transfer to Unseen Classes
We acknowledge that our previous results do not fully establish transfer to unseen problem families. Indeed, integrality gap prediction is evaluated on MILP-Evolve, the same dataset used for pretraining, with an instance-level rather than family-level split. Furthermore, seven D-MIPLIB families were used for pretraining hyperparameter selection and therefore cannot be considered entirely unseen at the model-selection level, even though they were excluded from encoder pretraining.
We would first like to clarify that the hyperparameter tuning procedure was introduced primarily to ensure a competitive comparison with FORGE†, rather than because ReMILP requires extensive tuning.
Indeed, ReMILP exhibits remarkably low sensitivity to hyperparameter selection. Across 54 completed Optuna trials, its downstream validation objective ranges from 0.7743 to 0.8375 (median: 0.7896, IQR: 0.0086). In contrast, across 56 FORGE† trials, the same objective ranges from 0.9471 to 3.4689 (IQR: 0.5592). Thus, FORGE† exhibits a 65× larger interquartile range, and even ReMILP's worst configuration outperforms FORGE†'s best configuration.
These results suggest that ReMILP's advantage is robust to the explored hyperparameter choices, rather than being driven by a particularly favorable configuration selected using downstream labels.
To provide evidence of generalization independently of model selection, we added three new problem families excluded entirely from encoder pretraining and pretraining hyperparameter selection: araujo2015 (Multi-Item Lot Sizing), bollapragada2001 (Structural Member Sizing) and zetina2019 (Network Design). All three were made using the generators provided by FrontierOR [1] and solved using Gurobi like the other classes. We evaluate retrieval, as well as frozen and fine-tuned solution prediction on these three classes, and the results confirm that our representations transfer cleanly to unseen classes.
Invariance and equivariance evaluation
These three real-world families do not exhibit the symmetry present in some of D-MIPLIB's synthetic problem classes. As such, they provide even stronger evidence that our representations carry the inductive biases we intended, with an equivariance Top-1 accuracy of and an equivariance MRR of .
Model | Invariance Top-1 () | Invariance MRR () | Equivariance Top-1 () | Equivariance MRR ()
- | - | - | - | - Random | 0.116 0.010 | 0.180 0.011 | 0.033 0.008 | 0.048 0.007 FORGE† | 0.130 0.033 | 0.196 0.036 | 0.030 0.003 | 0.041 0.003 Official FORGE | 0.146 0.001 | 0.192 0.000 | 0.003 0.000 | 0.014 0.000 ReMILP | 0.374 0.052 | 0.459 0.052 | 0.050 0.007 | 0.063 0.007 ReMILP + hyper | - | - | 0.439 0.019 | 0.532 0.017
Frozen encoders
Training a head for solution prediction on frozen ReMILP representations over 50k steps leads to an 8.8% improvement over training one on a randomly initialized GNN on average. FORGE† leads to a 0.9% improvement over Random, and the official FORGE checkpoint's representations lead to a 20% worse KL than our randomly initialized GNN.
Problem class | Random (Frozen) | FORGE† (Frozen) | Official FORGE (Frozen) | ReMILP (Ours, Frozen)
- | - | - | - | -
araujo2015| 0.6418 0.0050 | 0.6329 0.0210 | 0.6865 0.0000 | 0.5143 0.0056bollapragada2001| 0.2537 0.0046 | 0.2503 0.0015 | 0.3005 0.0000 | 0.2403 0.0041zetina2019| 0.2639 0.0000 | 0.2639 0.0001 | 0.3599 0.0000 | 0.2633 0.0002
Fine-tuned encoders
Once we fine-tune each of our models for 50k steps on these 3 classes, we observe that ReMILP retains a improvement over a model that is supervised from scratch, compared to for FORGE†. Fine-tuned official FORGE has, on average, 15.1% higher KL than our supervised baseline, further suggesting that its architecture may limit performance on these tasks.
| Problem Class | Supervised | FORGE† | Official FORGE | ReMILP |
|---|---|---|---|---|
araujo2015 | 0.4682 0.0182 | 0.4762 0.0094 | 0.6698 0.0004 | 0.4640 0.0013 |
bollapragada2001 | 0.2222 0.0003 | 0.2337 0.0092 | 0.2366 0.0007 | 0.2237 0.0005 |
zetina2019 | 0.2640 0.0001 | 0.2301 0.0014 | 0.2641 0.0001 | 0.2291 0.0016 |
[1] Kong, Minwei, et al. "FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization.", NeurIPS 2026 Evaluation & Dataset Track
Revised manuscript
Dear Reviewer K9Su,
We have uploaded a revised version of our manuscript (with changes highlighted in blue) integrating our rebuttal additions, including the invariant contrastive baselines (Appendix D.2), the hyperparameter sensitivity analysis (Appendix F.6), and the unseen out-of-distribution problem families (Appendix C).
As the discussion period is coming to an end in two days, we wanted to make sure you saw the revised paper and check if you have had a chance to go through our responses. Please let us know if your concerns have been addressed. We would be happy to clarify any remaining points or answer further questions.
Thank you very much!
Invariant contrastive model comparison
We thank the reviewer for their valuable feedback.
W1 & Q1 : Invariant Contrastive Model Ablation
For the contrastive baseline, to directly test whether a standard contrastive GNN on the Gasse bipartite representation suffices, we trained two generic invariant contrastive baselines on the exact same GNN encoder (d=128, 2-layer GATv2) and identical pretraining budget (20k steps, 3 seeds):
Below is the comparison on downstream D-MIPLIB tasks (relative geomean ↓) as well as Integrality Gap prediction (MAE ↓):
We find that the Contrastive (With Substitutions) matches ReMILP on activity prediction and integrality gap prediction, and underperforms on MILP solution prediction. This makes intuitive sense as constraint activity is invariant to substitutions, whereas variable assignments aren't, hence the value of the equivariance machinery in ReMILP.