acceptodds
Under review as a conference paper at ICLR 2027

Reformulation-Contrastive Learning for Mixed Integer Programs

Abstract

Mixed-integer linear programs (MILP) model many real-world decision problems, motivating machine-learning methods that exploit recurring structure to accelerate MILP solving. MILPs can admit many equivalent formulations: integrality-preserving changes of variables and the addition of redundant constraints can alter their formulations while preserving the optimization problem. We leverage these reformulations as a source of self-supervision for learning general-purpose representations of MILP variables and constraints. We characterize the affine reformulations that are valid for every input instance, and distinguish re-descriptions, which leave variables unchanged, from substitutions, which transform them predictably. Building on equivariant self-supervised learning, we introduce ReMILP (reformulation-contrastive MILP representation learning), which jointly trains a graph neural network and a hypernetwork to predict how variable embeddings transform under changes of variables. Without solver-derived labels, ReMILP learns representations that exhibit the intended invariance and equivariance on unseen problem classes. Across binary solution, constraint activity and integrality gap prediction, these representations carry task-relevant information when frozen and provide a useful initialization for fine-tuning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Mixed-integer programming, Representation learning, Self-supervised learning, Contrastive learning, Graph neural network

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_K9SuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies self-supervised MILP representation learning using equivalent reformulations. It characterizes universally valid affine augmentation rules and separates constraint re-descriptions from variable substitutions. ReMILP combines a graph encoder with a hypernetwork that predicts changes in variable embeddings under substitutions. Experiments evaluate reformulation retrieval and transfer to binary solution, constraint activity, and integrality-gap prediction, using both frozen and fine-tuned encoders.

Critical strengths

The distinction between re-descriptions and substitutions gives the learning objective a concrete motivation. Making all equivalent formulations invariant would ignore the fact that a change of variables also changes what each variable represents. The characterization in Section 4.1 is useful in its own right, and the paper is explicit about its restriction to affine rules that work for every instance.

There is evidence that the representations contain useful information beyond what a random encoder provides. The frozen integrality-gap result is particularly encouraging: MAE falls from 0.063 for Random and 0.053 for FORGE to 0.040 for ReMILP. The shared backbone and the additional FORGE pooling comparison help make this result interpretable. Appendix B also provides a useful comparison against pre-training with redundant constraints alone.

Critical weaknesses

The experiments do not yet isolate the value of learning an equivariant action. Appendix B removes substitutions, the hypernetwork, and the action negatives together. It establishes a benefit from the full substitution-based setup over re-description-only pre-training, but does not establish whether a standard invariant contrastive objective on the same substituted views would perform worse. This matters because the invariance/equivariance distinction is the main methodological argument. Table 1 shows improved retrieval with the hypernetwork, but that is a closely related pre-training diagnostic, and its Top-1 accuracy is still 0.162. It does not resolve whether learning the action is what improves downstream prediction. A comparison using identical reformulated views but an invariant objective would be more decisive.

The scope of the transfer evidence needs more care. Section D.6 selects pre-training hyperparameters through 56 trials per method, using labeled downstream validation performance on seven D-MIPLIB families. Thus, although the encoder loss uses no solver labels, the complete selection procedure does. Table 1 then aggregates retrieval on all 23 families, including those used for selection, under the heading of unseen classes. This is not test-set leakage, but it is weaker evidence of class-level generalization than the wording suggests. MILP-Evolve is also split randomly by instance (Section C.1), rather than by family. Results separated into selection families and untouched problem classes would help establish how broadly the learned prior transfers.

Questions for authors
  • How does an invariant contrastive model perform when trained on exactly the same substituted and re-described views as ReMILP? Appendix B changes the augmentation distribution as well as the objective, so it cannot answer this question.
  • Could you report retrieval and downstream results separately for problem classes used in hyperparameter selection and classes excluded from selection? For MILP-Evolve, how much instance-family overlap is there across the pre-training and evaluation splits?
3 comments
AuthorsOpenReview ↗

Invariant contrastive model comparison

We thank the reviewer for their valuable feedback.

W1 & Q1 : Invariant Contrastive Model Ablation

For the contrastive baseline, to directly test whether a standard contrastive GNN on the Gasse bipartite representation suffices, we trained two generic invariant contrastive baselines on the exact same GNN encoder (d=128d=128, 2-layer GATv2) and identical pretraining budget (20k steps, 3 seeds):

  • Contrastive, Without Substitutions: Standard invariant contrastive objective over constraint re-descriptions alone, tested in appendix B.
  • Contrastive, With Substitutions: Standard invariant contrastive objective trained on the exact same substituted and re-described views as ReMILP, without learning an equivariant transformation or action predictor.

Below is the comparison on downstream D-MIPLIB tasks (relative geomean ↓\downarrow) as well as Integrality Gap prediction (MAE ↓\downarrow):

AblationSolution, MILPSolution, ILPActivity, MILPActivity, ILPIntegrality Gap
Supervised1.0001.0001.0001.0000.0454 ±\pm 0.0040
Contrastive, Without Substitutions1.004 ±\pm 0.0191.002 ±\pm 0.0120.997 ±\pm 0.0350.998 ±\pm 0.0040.0418 ±\pm 0.0024
Contrastive, With Substitutions0.917 ±\pm 0.0250.996 ±\pm 0.0050.897 ±\pm 0.0530.992 ±\pm 0.0010.0388 ±\pm 0.0035
FORGE† (Reimplementation)0.911 ±\pm 0.0180.992 ±\pm 0.0030.985 ±\pm 0.0420.992 ±\pm 0.0030.0468 ±\pm 0.0021
ReMILP (Ours)0.854 ±\pm 0.0200.988 ±\pm 0.0130.918 ±\pm 0.0510.991 ±\pm 0.0030.0378 ±\pm 0.0014

We find that the Contrastive (With Substitutions) matches ReMILP on activity prediction and integrality gap prediction, and underperforms on MILP solution prediction. This makes intuitive sense as constraint activity is invariant to substitutions, whereas variable assignments aren't, hence the value of the equivariance machinery in ReMILP.

AuthorsOpenReview ↗

Hyperparameter Tuning and Transfer to Unseen Classes

W2 & Q2: Hyperparameter Tuning and Transfer to Unseen Classes

We acknowledge that our previous results do not fully establish transfer to unseen problem families. Indeed, integrality gap prediction is evaluated on MILP-Evolve, the same dataset used for pretraining, with an instance-level rather than family-level split. Furthermore, seven D-MIPLIB families were used for pretraining hyperparameter selection and therefore cannot be considered entirely unseen at the model-selection level, even though they were excluded from encoder pretraining.

We would first like to clarify that the hyperparameter tuning procedure was introduced primarily to ensure a competitive comparison with FORGE†, rather than because ReMILP requires extensive tuning.

Indeed, ReMILP exhibits remarkably low sensitivity to hyperparameter selection. Across 54 completed Optuna trials, its downstream validation objective ranges from 0.7743 to 0.8375 (median: 0.7896, IQR: 0.0086). In contrast, across 56 FORGE† trials, the same objective ranges from 0.9471 to 3.4689 (IQR: 0.5592). Thus, FORGE† exhibits a 65× larger interquartile range, and even ReMILP's worst configuration outperforms FORGE†'s best configuration.

These results suggest that ReMILP's advantage is robust to the explored hyperparameter choices, rather than being driven by a particularly favorable configuration selected using downstream labels.

To provide evidence of generalization independently of model selection, we added three new problem families excluded entirely from encoder pretraining and pretraining hyperparameter selection: araujo2015 (Multi-Item Lot Sizing), bollapragada2001 (Structural Member Sizing) and zetina2019 (Network Design). All three were made using the generators provided by FrontierOR [1] and solved using Gurobi like the other classes. We evaluate retrieval, as well as frozen and fine-tuned solution prediction on these three classes, and the results confirm that our representations transfer cleanly to unseen classes.

Invariance and equivariance evaluation

These three real-world families do not exhibit the symmetry present in some of D-MIPLIB's synthetic problem classes. As such, they provide even stronger evidence that our representations carry the inductive biases we intended, with an equivariance Top-1 accuracy of 43.943.9\\% and an equivariance MRR of 53.253.2\\%.

Model | Invariance Top-1 (↑\uparrow) | Invariance MRR (↑\uparrow) | Equivariance Top-1 (↑\uparrow) | Equivariance MRR (↑\uparrow)

  • | - | - | - | - Random | 0.116 ±\pm 0.010 | 0.180 ±\pm 0.011 | 0.033 ±\pm 0.008 | 0.048 ±\pm 0.007 FORGE† | 0.130 ±\pm 0.033 | 0.196 ±\pm 0.036 | 0.030 ±\pm 0.003 | 0.041 ±\pm 0.003 Official FORGE | 0.146 ±\pm 0.001 | 0.192 ±\pm 0.000 | 0.003 ±\pm 0.000 | 0.014 ±\pm 0.000 ReMILP | 0.374 ±\pm 0.052 | 0.459 ±\pm 0.052 | 0.050 ±\pm 0.007 | 0.063 ±\pm 0.007 ReMILP + hyper | - | - | 0.439 ±\pm 0.019 | 0.532 ±\pm 0.017

Frozen encoders

Training a head for solution prediction on frozen ReMILP representations over 50k steps leads to an 8.8% improvement over training one on a randomly initialized GNN on average. FORGE† leads to a 0.9% improvement over Random, and the official FORGE checkpoint's representations lead to a 20% worse KL than our randomly initialized GNN.

Problem class | Random (Frozen) | FORGE† (Frozen) | Official FORGE (Frozen) | ReMILP (Ours, Frozen)

  • | - | - | - | - araujo2015 | 0.6418 ±\pm 0.0050 | 0.6329 ±\pm 0.0210 | 0.6865 ±\pm 0.0000 | 0.5143 ±\pm 0.0056 bollapragada2001 | 0.2537 ±\pm 0.0046 | 0.2503 ±\pm 0.0015 | 0.3005 ±\pm 0.0000 | 0.2403 ±\pm 0.0041 zetina2019 | 0.2639 ±\pm 0.0000 | 0.2639 ±\pm 0.0001 | 0.3599 ±\pm 0.0000 | 0.2633 ±\pm 0.0002

Fine-tuned encoders

Once we fine-tune each of our models for 50k steps on these 3 classes, we observe that ReMILP retains a 4.74.7\\% improvement over a model that is supervised from scratch, compared to 2.32.3\\% for FORGE†. Fine-tuned official FORGE has, on average, 15.1% higher KL than our supervised baseline, further suggesting that its architecture may limit performance on these tasks.

Problem ClassSupervisedFORGE†Official FORGEReMILP
araujo20150.4682 ±\pm 0.01820.4762 ±\pm 0.00940.6698 ±\pm 0.00040.4640 ±\pm 0.0013
bollapragada20010.2222 ±\pm 0.00030.2337 ±\pm 0.00920.2366 ±\pm 0.00070.2237 ±\pm 0.0005
zetina20190.2640 ±\pm 0.00010.2301 ±\pm 0.00140.2641 ±\pm 0.00010.2291 ±\pm 0.0016

[1] Kong, Minwei, et al. "FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization.", NeurIPS 2026 Evaluation & Dataset Track

AuthorsOpenReview ↗

Revised manuscript

Dear Reviewer K9Su,

We have uploaded a revised version of our manuscript (with changes highlighted in blue) integrating our rebuttal additions, including the invariant contrastive baselines (Appendix D.2), the hyperparameter sensitivity analysis (Appendix F.6), and the unseen out-of-distribution problem families (Appendix C).

As the discussion period is coming to an end in two days, we wanted to make sure you saw the revised paper and check if you have had a chance to go through our responses. Please let us know if your concerns have been addressed. We would be happy to clarify any remaining points or answer further questions.

Thank you very much!

Reviewer_rVaAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReMILP (reformulation-contrastive MILP representation learning), which uses equivalent reformulations of a MILP instance as a self-supervision signal for representation learning. The paper proves that every valid affine augmentation rule is a composition of a re-description and a substitution. Building on this, a GNN encoder and a hypernetwork are trained jointly with a contrastive loss so that variable embeddings are invariant to re-descriptions and equivariant to substitutions. ReMILP is evaluated on three tasks (binary-solution prediction, constraint-activity prediction, and integrality-gap prediction).

Critical strengths

The distinction between invariance to re-descriptions and equivariance to substitutions is reasonable, and using it as a self-supervision signal for MILP representation learning appears to be new. The approach is supported by a theoretical characterization of the affine reformulation rules that are valid for every instance, which goes beyond the shifts and signed permutations used in prior work.

Critical weaknesses
  1. The reported improvements are not shown to translate into actual solving performance. The paper evaluates on three downstream tasks (integrality-gap prediction, binary-solution prediction, and constraint-activity prediction), but it does not establish a connection between these tasks and actual solving performance. The authors of [1] describe the integrality gap only as a "potential indicator" of solving time, and they additionally include learning-to-branch as a task to demonstrate solving efficiency. Likewise, FORGE [2] demonstrates that its embeddings improve solver performance when used to guide Gurobi's search. For the binary-solution and constraint-activity tasks, the solution-pool weighting follows [3]; however, in [3] this weighting is used only to define the training loss, not as an evaluation measure. The reported metrics (KL, MAE) for the chosen tasks do not directly measure solving performance and should therefore be regarded as proxies. The paper offers no evidence that gains on these proxies translate into solver performance.

  2. The baselines are weak. The only baseline taken from the existing literature is FORGE, and even that is the authors' own re-implementation rather than the original model. While sharing a backbone helps isolate the effect of the pre-training objective, it does not establish how ReMILP compares with FORGE in its original configuration. Given that the FORGE authors released their code, results with the original configuration should also be reported. In addition, related representation-learning methods [4, 5], which are discussed in Section 2, are excluded from the experimental comparison without any stated reason. The paper notes that these methods rely on solver-derived labels; however, Appendix D.6 shows that ReMILP's pre-training hyperparameters are selected based on downstream validation performance on tasks that require solver-derived labels. Although the pre-training objective itself is label-free, the overall training pipeline, which includes hyperparameter tuning, is not. This weakens the paper's label-free framing and makes the omission of supervised representation-learning baselines [4, 5] less convincing.

[1] Towards foundation models for mixed integer linear programming, ICLR, 2025.

[2] FORGE: Foundational Optimization Representations from Graph Embeddings, TMLR, 2026.

[3] Solving Mixed Integer Programs Using Neural Networks, arXiv, 2020.

[4] Multi-task representation learning for mixed integer linear programming, CPAIOR, 2025.

[5] RoME: Domain-robust mixture-of-experts for MILP solution prediction across domains, NeurIPS, 2025.

Questions for authors
  1. The seven instance families used to tune ReMILP's pre-training hyperparameters (Line 1211) come from seven of the ten D-MIPLIB problem classes, so these classes are not strictly unseen at the model-selection level. Could you report Tables 1 and 2 separately for the three classes not used in tuning (IS, VC, GISP) and the seven classes used in tuning?

  2. Could you evaluate predictions from at least one downstream task within an actual solver pipeline (e.g., binary-solution predictions in a predict-and-search framework [6]) and report primal gap or solving time against the baselines?

  3. How exactly does FORGE† differ from the original FORGE configuration (e.g., the original uses GraphSAGE layers)? Have you compared ReMILP against FORGE in its original configuration?

  4. Keeping the total number of negatives in Eq. (12) fixed, what are the results when each type of negative is removed in turn, and what insights can be drawn from this?

[6] A GNN-Guided Predict-and-Search Framework for Mixed-Integer Linear Programming, ICLR, 2023.

9 comments
AuthorsOpenReview ↗

Solver evaluation

We thank the reviewer for their careful assessment and constructive feedback.

W1 & Q2: Solver evaluation

We have integrated our fine-tuned representations into Gurobi using the Predict-and-Search (P&S) framework (Han et al., 2023) across four D-MIPLIB classes: MMCN-medium-BC, OTS-hard, CFLP-medium, and LB-hard, and measured the resulting wall-clock time, number of explored nodes, final primal-dual gap, and primal integral (integral of the gap with respect to time).

ReMILP advantage over the other learning models is clearest on MMCN-medium-BC and OTS-hard, where it achieves substantially smaller final gaps. Compared with default Gurobi, it reduces runtime by 13.2% and primal integral by 17.5% on OTS-hard, retaining a zero final gap. On MMCN-medium-BC, it reduces runtime by a factor of 64 with a 0.077% final gap. On CFLP-medium and LB-hard, all learning methods have comparable final gaps.

These results are relatively consistent with what the binary solution prediction loss indicated, since ReMILP only had a significant lead on MMCN-medium-BC and OTS-hard, and all three methods were within one standard deviation of each other on LB-hard in terms of KL loss.

Problem Class | Method | Avg Runtime (s) | Avg Explored Nodes | Primal Gap (%) | Primal Integral

  • | - | - | - | - | - MMCN-medium-BC | Gurobi Default | 56.13s | 5,174 | 0.000% | 0.31 | | ReMILP (Ours) | 0.87s | 87 | 0.077% | 1.22 | | FORGE† | 0.57s | 13 | 10.173% | 102.10 | | Supervised | 0.52s | 16 | 12.103% | 121.38 OTS-hard | Gurobi Default | 67.20s | 731 | 0.000% | 48.38 | | ReMILP (Ours) | 58.31s | 719 | 0.000% | 39.89 | | FORGE† | 51.56s | 648 | 1.762% | 52.96 | | Supervised | 49.21s | 667 | 7.010% | 106.96 CFLP-medium | Gurobi Default | 7.97s | 942 | 0.000% | 0.34 | | ReMILP (Ours) | 4.93s | 626 | 0.029% | 0.52 | | FORGE† | 4.08s | 551 | 0.030% | 0.54 | | Supervised | 3.73s | 513 | 0.031% | 0.54 LB-hard | Gurobi Default | 900.68s | 15,020 | 0.006% | 5.59 | | ReMILP (Ours) | 909.20s | 13,777 | 0.003% | 7.06 | | FORGE† | 879.61s | 13,796 | 0.000% | 7.06 | | Supervised | 862.92s | 15,190 | 0.000% | 6.12

For this experiment, we used Gurobi 12.0 single-threaded on a Ryzen 9 9950X CPU, with a 1000s time limit per instance. Since all three models' highest-confidence predictions agreed, we had to choose less conservative values for the settings (k0,k1,Δ)(k_0, k_1, \Delta) of P&S to better distinguish their effects on solver performance. We set Δ=2\Delta = 2 for all four classes, (k0,k1)=(90(k_0, k_1) = (90\\%, 2\\%) for MMCN-medium-BC, (k0,k1)=(45(k_0, k_1) = (45\\%, 1\\%) for OTS-hard, (k0,k1)=(75(k_0, k_1) = (75\\%, 8\\%) for CFLP-medium, and (k0,k1)=(20(k_0, k_1) = (20\\%, 45\\%) for LB-hard. These settings were chosen based on the distribution of binary variable assignments in the training instances' solution pools, without using test-set performance.

AuthorsOpenReview ↗

Original FORGE configuration

W2 & Q3: Original FORGE configuration

The main difference between our re-implementation of FORGE and the original configuration is the architecture of the GNN encoder:

ComponentOur GNN encoderOriginal FORGE Encoder
Graph StructureHeterogeneous bipartite graphHomogeneous undirected graph
Input Node FeaturesVariable features and constraint features are separately projected into the latent space after a row normalization preprocessing stepEach variable and constraint is attributed a 10-dimensional zero-padded vector of features which is then min-max normalized over each graph before being projected into the latent space
Edge EmbeddingsEdges have 128-dimensional embeddingsScalar edge weights
Message Passing2 layers, each of which has 2 alternating bipartite GATv2 message-passing steps (variables to constraints then constraints to variables)2 GraphSAGE layers with mean aggregation
Node Embedding Dimension1281024
RegularizationNoneBatchNorm and Dropout after every layer
Layer AggregationJumping Knowledge (concatenates all intermediate representations of each node)Final layer output only
Total number of Parameters0.66M3.18M, ~4.8× larger

We addressed the concern regarding our FORGE baseline by evaluating the official pretrained FORGE checkpoint released by Shafi et al. on Hugging Face under both frozen and fine-tuned regimes, across all 46 node-level tasks (relative geomean ↓\downarrow) and integrality gap prediction (MAE ↓\downarrow). In both settings, the official model performs substantially worse than our FORGE† reimplementation and is consistently worse than our randomly initialized GNN baseline. Pretraining: ReMILP/FORGE† on MILP-Evolve, official FORGE on MIPLIB+D-MIPLIB. Evaluation: solution/activity on D-MIPLIB, gap on MILP-Evolve.

Frozen encoders

Task GroupRandom (Frozen)FORGE† (Frozen)Official FORGE (Frozen)ReMILP (Frozen)
Binary solution, ILP1.0000.918 ±\pm 0.0071.1460.887 ±\pm 0.016
Binary solution, MILP1.0000.755 ±\pm 0.0121.3670.688 ±\pm 0.050
Constraint activity, ILP1.0000.882 ±\pm 0.0181.2220.871 ±\pm 0.007
Constraint activity, MILP1.0000.863 ±\pm 0.0241.8310.855 ±\pm 0.028
Integrality gap0.0634 ±\pm 0.00840.0530 ±\pm 0.00660.21890.0398 ±\pm 0.0019

Fine-tuned encoders

Task GroupSupervisedFORGE†Official FORGEReMILP
Binary solution, ILP1.0000.994 ±\pm 0.0041.2561.001 ±\pm 0.009
Binary solution, MILP1.0000.919 ±\pm 0.0651.4540.850 ±\pm 0.024
Constraint activity, ILP1.0000.999 ±\pm 0.0041.3400.994 ±\pm 0.004
Constraint activity, MILP1.0000.983 ±\pm 0.0301.7410.959 ±\pm 0.018
Integrality gap0.0426 ±\pm 0.00370.0468 ±\pm 0.00550.23440.0366 ±\pm 0.0014
AuthorsOpenReview ↗

Hyperparameter tuning and transfer to unseen classes

W2 & Q1: Hyperparameter Tuning and OOD Transfer

Regarding the omission of supervised representation-learning baselines [4, 5], we emphasize that these methods operate in a different supervision setting. Cai et al. [4] learn representations through multiple solver-supervised tasks, while RoME [5] trains specifically for binary-solution prediction using solver-generated solutions. In contrast, ReMILP pretrains a task-agnostic encoder exclusively from equivalent reformulations, without solver labels. We therefore focus on self-supervised baselines to isolate the effectiveness of our pretraining objective, and exclude methods that require task-specific solver supervision from this comparison. Importantly, the use of downstream labels for hyperparameter selection does not change this distinction. We introduced this procedure primarily to ensure a competitive comparison with FORGE†, rather than because ReMILP requires extensive tuning.

Indeed, ReMILP exhibits remarkably low sensitivity to hyperparameter selection. Across 54 completed Optuna trials, its downstream validation objective ranges from 0.7743 to 0.8375 (IQR: 0.0086). In contrast, across 56 FORGE† trials, the same objective ranges from 0.9471 to 3.4689 (IQR: 0.5592). Thus, FORGE† exhibits a 65× larger interquartile range, and even ReMILP's worst configuration outperforms FORGE†'s best configuration. These results suggest that ReMILP's advantage is robust to the explored hyperparameter choices, rather than being driven by a particularly favorable configuration selected using downstream labels.

Nevertheless, we acknowledge that using downstream validation data for model selection weakens claims of strictly out-of-distribution evaluation on the corresponding problem families. We therefore added three new problem families excluded entirely from encoder pretraining and hyperparameter selection: araujo2015 (Multi-Item Lot Sizing), bollapragada2001 (Structural Member Sizing) and zetina2019 (Network Design). All three were made using the generators provided by FrontierOR [1] and solved using Gurobi like the other classes. We evaluate retrieval, as well as frozen and fine-tuned solution prediction on these new classes, and the results confirm that our representations transfer cleanly to unseen classes.

Invariance and equivariance evaluation

These three real-world families do not exhibit the symmetry present in some of D-MIPLIB's synthetic problem classes. As such, they provide even stronger evidence that our representations carry the inductive biases we intended, with an equivariance Top-1 accuracy of 43.943.9\\% and an equivariance MRR of 53.253.2\\%.

Model | Invariance Top-1 (↑\uparrow) | Invariance MRR (↑\uparrow) | Equivariance Top-1 (↑\uparrow) | Equivariance MRR (↑\uparrow)

  • | - | - | - | - Random | 0.116 ±\pm 0.010 | 0.180 ±\pm 0.011 | 0.033 ±\pm 0.008 | 0.048 ±\pm 0.007 FORGE† | 0.130 ±\pm 0.033 | 0.196 ±\pm 0.036 | 0.030 ±\pm 0.003 | 0.041 ±\pm 0.003 Official FORGE | 0.146 ±\pm 0.001 | 0.192 ±\pm 0.000 | 0.003 ±\pm 0.000 | 0.014 ±\pm 0.000 ReMILP | 0.374 ±\pm 0.052 | 0.459 ±\pm 0.052 | 0.050 ±\pm 0.007 | 0.063 ±\pm 0.007 ReMILP + hyper | - | - | 0.439 ±\pm 0.019 | 0.532 ±\pm 0.017

Frozen encoders

Training a head for solution prediction on frozen ReMILP representations over 50k steps leads to an 8.8% improvement over training one on a randomly initialized GNN on average. FORGE† leads to a 0.9% improvement over Random, and the official FORGE checkpoint's representations lead to a 20% worse KL than our randomly initialized GNN.

class | Random (Frozen) | FORGE† (Frozen) | Official FORGE (Frozen) | ReMILP (Ours, Frozen)

  • | - | - | - | - araujo2015 | 0.6418 ±\pm 0.0050 | 0.6329 ±\pm 0.0210 | 0.6865 ±\pm 0.0000 | 0.5143 ±\pm 0.0056 bollapragada2001 | 0.2537 ±\pm 0.0046 | 0.2503 ±\pm 0.0015 | 0.3005 ±\pm 0.0000 | 0.2403 ±\pm 0.0041 zetina2019 | 0.2639 ±\pm 0.0000 | 0.2639 ±\pm 0.0001 | 0.3599 ±\pm 0.0000 | 0.2633 ±\pm 0.0002

Fine-tuned encoders

Once we fine-tune each of our models for 50k steps on these 3 classes, we observe that ReMILP retains a 4.74.7\\% improvement over a model that is supervised from scratch, compared to 2.32.3\\% for FORGE†. Fine-tuned official FORGE has, on average, 15.1% higher KL than our supervised baseline, further suggesting that its architecture may limit performance on these tasks.

ClassSupervisedFORGE†Official FORGEReMILP
araujo20150.4682 ±\pm 0.01820.4762 ±\pm 0.00940.6698 ±\pm 0.00040.4640 ±\pm 0.0013
bollapragada20010.2222 ±\pm 0.00030.2337 ±\pm 0.00920.2366 ±\pm 0.00070.2237 ±\pm 0.0005
zetina20190.2640 ±\pm 0.00010.2301 ±\pm 0.00140.2641 ±\pm 0.00010.2291 ±\pm 0.0016

[1] Kong, Minwei, et al. "FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization.", NeurIPS 2026

Reviewer_WktjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a contrastive learning approach to learn equivalent representations of MIPs.

Critical strengths

The paper proposes a contrastive learning approach specific to MIPs, which is shown to empirically outperform a representation on its specific task.

Critical weaknesses

The paper only compares against another baseline, FORGE. The presented experiments assess the performance specifically on the same loss as the training of the model, which is different from the one FORGE uses. The paper never assesses whether the proposed approach would outperform a generic contrastive learning approach based on GNNs with the standard bipartite MIP representation of Gasse, which would be a strong baseline for this problem.

Minor

To reduce the reading complexity of the paper, it seems unnecessary to introduce both concepts of re-writing and re-description, since one is the valid case for the more general re-writing. Re-writing just appears when introducing the definition and re-description is used throughout.

Mathematical clarity should be improved. What is the meaning of (7), is it a constraint added on the model for every valid (Λ,g)(\Lambda, g)?

Questions for authors

How are the properties introduced in 4.2 an inductive bias? How is it encouraged in training?

2 comments
AuthorsOpenReview ↗

Method clarifications and contrastive baselines

We thank the reviewer for their comments and suggestions.

W1: Baseline

Both our method (ReMILP) and FORGE† are pretrained with their own respective loss functions. Ours is a modified contrastive loss, whereas FORGE's is a vector-quantized reconstruction loss. After that, both models are fine-tuned on the same downstream tasks with the same loss functions, and evaluated based on their performance on those tasks. As such, the only difference between the two fine-tuned models is their initialization, and neither ReMILP nor FORGE is evaluated on its pretraining objective. The retrieval experiment in Section 5.2.1 assesses whether the ReMILP representations exhibit the intended invariance and equivariance properties. Their usefulness for downstream prediction is evaluated separately in Sections 5.2.2 and 5.2.3.

For the contrastive baseline, to directly test whether a standard contrastive GNN on the Gasse bipartite representation suffices, we trained two generic invariant contrastive baselines on the exact same GNN encoder (d=128d=128, 2-layer GATv2) and identical pretraining budget (20k steps, 3 seeds):

  • Contrastive, Without Substitutions: Standard invariant contrastive objective over constraint re-descriptions alone (no variable substitutions).
  • Contrastive, With Substitutions: Standard invariant contrastive objective trained on the exact same substituted and re-described views as ReMILP, without learning an equivariant transformation or action predictor.

Below is the comparison on downstream D-MIPLIB tasks (relative geomean ↓\downarrow) as well as Integrality Gap prediction (MAE ↓\downarrow):

AblationSolution, MILPSolution, ILPActivity, MILPActivity, ILPIntegrality Gap
Supervised1.0001.0001.0001.0000.0454 ±\pm 0.0040
Contrastive, Without Substitutions1.004 ±\pm 0.0191.002 ±\pm 0.0120.997 ±\pm 0.0350.998 ±\pm 0.0040.0418 ±\pm 0.0024
Contrastive, With Substitutions0.917 ±\pm 0.0250.996 ±\pm 0.0050.897 ±\pm 0.0530.992 ±\pm 0.0010.0388 ±\pm 0.0035
FORGE† (Reimplementation)0.911 ±\pm 0.0180.992 ±\pm 0.0030.985 ±\pm 0.0420.992 ±\pm 0.0030.0468 ±\pm 0.0021
ReMILP (Ours)0.854 ±\pm 0.0200.988 ±\pm 0.0130.918 ±\pm 0.0510.991 ±\pm 0.0030.0378 ±\pm 0.0014

We find that the Contrastive (Without Substitutions) is virtually equivalent to Supervised on all tasks except Integrality Gap Prediction, and that it is outmatched by ReMILP on every task. This establishes that substitutions play an important role in ReMILP. We also find that the Contrastive (With Substitutions) matches ReMILP on activity prediction and integrality gap prediction, and underperforms on MILP solution prediction. This makes intuitive sense as constraint activity is invariant to substitutions, whereas variable assignments aren't, hence the value of the equivariance machinery in ReMILP.

Q1: Inductive Biases

The idea behind section 4.2 is to make our learned representations respect the mathematical symmetries of MILPs without relying on solver labels, at least in an approximate sense. Equation (7) says that, ideally, adding redundant constraints shouldn't change our variable embeddings (and predictions thereof). Equation (8) says that applying a change of variables should change our variable embeddings in a predictable way. We do not enforce these properties strictly, but we encourage them through our modified contrastive loss in Equation (13) during pretraining.

Minor : Confusing Terminology

We acknowledge that the terminology we use is a bit confusing. We will simplify the terminology by removing "constraint rewriting" as a separate term. We will define re-descriptions directly as constraint transformations of the form in Eq. (5) that preserve every instance’s feasible set. We will retain the definition of R(Λ,g)R(\Lambda,g) in Eq. (5) and replace "constraint rewriting" with "the map R(Λ,g)R(\Lambda,g)" in Theorem 4.2 and the corresponding appendix passages.

AuthorsOpenReview ↗

Revised manuscript

Dear Reviewer Wktj,

We have uploaded a revised version of our manuscript (with changes highlighted in blue) incorporating our rebuttal updates, including the invariant contrastive pretraining baselines (Appendix D.2) and the simplified terminology around re-descriptions in Section 4.1.

As the discussion period is coming to an end in two days, we wanted to make sure you saw the revised paper and check if you have had a chance to go through our responses. Please let us know if your concerns have been addressed. We would be happy to clarify any remaining points or answer further questions.

Thank you very much!

Reviewer_UEpqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A mixed integer linear program (MILP) can be written in many equivalent ways. The same problem, rescaled or rewritten, is still the same problem. The paper asks: what are all the rewritings (affine augmentations) that are valid for every MILP? It answers with three theorems. Every valid affine rule splits into two parts. A re description recombines constraints with nonnegative weights: it must keep a positively rescaled copy of each constraint, and it may loosen right hand sides. A substitution is an affine change of variables whose continuous part is invertible and whose integer part is unimodular. This is more general than the shifts and sign flips used in earlier work. The appendix gives full proofs.

ReMILP turns this theory into a self supervised pre training task. A graph neural network encodes each variable. Its embeddings should be invariant to re descriptions (the variables did not change) and equivariant to substitutions (they change in a predictable way). A hypernetwork predicts how a substitution transforms each embedding, and a contrastive loss (NT Xent, with two kinds of negatives) enforces both rules. No solver labels are needed. On problem classes not seen in training, ReMILP retrieves variables correctly under both rewriting types, better than FORGE (the authors' re implementation of the only prior self supervised MILP representation method). Frozen embeddings beat FORGE on integrality gap prediction (minus 24.5% error) and binary solution prediction (minus 3.4% KL on pure ILP families, minus 8.9% on mixed families). Fine tuning mostly speeds up training on pure ILPs but keeps a lasting advantage on mixed integer families (integrality gap +16% over training from scratch).

Critical strengths
  1. A clean completeness theorem. The paper characterizes all valid affine MILP augmentations and corrects earlier work: integer variables allow arbitrary unimodular changes (including transvections), not only shifts and sign flips. The proofs are careful, and the re description proof uses Farkas' lemma properly.
  2. Theory matches the objective. The two theorem types map exactly onto the two training signals (invariance and equivariance), so the inductive bias is not decoration. The retrieval test on held out problem classes verifies that the intended semantics were learned.
  3. Negative results are reported honestly. Constraint activity prediction is statistically indistinguishable from FORGE. On pure ILPs, pre training only speeds up fine tuning and does not improve the final number. A no substitution ablation (Appendix B) cleanly shows the lasting gains come from the equivariance machinery.
  4. Solver label free pre training, evaluated on three downstream tasks. Processed datasets and a reproducible environment are promised.
  5. Positioning is fair against FORGE, multi task MILP encoders, and prior contrastive MILP branching work.
Critical weaknesses
  1. No solver level evaluation. The motivation is faster MILP solving, but no experiment measures solving: no branching integration, no predict and search, no wall clock comparison. For a top venue this is the main missing piece.
  2. The central baseline is the authors' own re implementation. FORGE is concurrent or under review work, re implemented and pre trained by the authors for a fixed 20,000 steps. If the reproduction is off, the main comparison is off. At minimum, configs and a sensitivity check on FORGE's pre training compute are needed.
  3. Theory is broader than practice. The theorems cover arbitrary valid substitution matrices, but training only uses local rank one substitutions: one variable plus one partner. The paper never ablates the richness of the substitution family, so it is unclear how much of the theorem's generality is actually used.
  4. "Equivariance" is soft. The action predictor ρ is not required to be a group homomorphism or invertible, so equivariance exists only as a contrastive target, not as a structural property of the encoder. Retrieval Top 1 is a coarse way to verify it.
  5. Single source pre training distribution. The integrality gap test stays in domain (held out MILP Evolve), and transfer to genuinely different problem families is only partially tested.
Questions for authors
  1. Can you plug the representation (frozen or fine tuned) into a real solver component and report nodes to solution or wall clock on benchmark families?
  2. Is the gap to FORGE stable if FORGE is pre trained longer or with more negatives?
  3. How sensitive is downstream transfer to the substitution family: number of partners, λ range, share of integer transvections?
  4. How much of the equivariance result survives if the partner embedding is removed from the hypernetwork input?
  5. What is the training time cost of re descriptions, since they can densify the graph?
4 comments
AuthorsOpenReview ↗

Solver evaluation and FORGE comparison

We thank the reviewer for their thoughtful assessment and constructive suggestions, which helped us strengthen the experimental evaluation.

W1 & Q1: Solver evaluation

We have integrated our fine-tuned representations into Gurobi using the Predict-and-Search (P&S) framework (Han et al., 2023) across four D-MIPLIB classes: MMCN-medium-BC, OTS-hard, CFLP-medium, and LB-hard, and measured the resulting wall-clock time, number of explored nodes, final primal-dual gap, and primal integral (integral of the gap with respect to time).

ReMILP advantage over the other learning models is clearest on MMCN-medium-BC and OTS-hard, where it achieves substantially smaller final gaps. Compared with default Gurobi, it reduces runtime by 13.2% and primal integral by 17.5% on OTS-hard, retaining a zero final gap. On MMCN-medium-BC, it reduces runtime by a factor of 64 at the cost of a 0.077% gap. On CFLP-medium and LB-hard, all learning methods have comparable gaps.

These results are consistent with what the binary solution prediction loss indicated, since ReMILP only had a significant lead on MMCN-medium-BC and OTS-hard, and all three methods were within one standard deviation of each other on LB-hard.

Problem Class | Method | Avg Runtime (s) | Avg Explored Nodes | Primal Gap (%) | Primal Integral

  • | - | - | - | - | - MMCN-medium-BC | Gurobi Default | 56.13s | 5,174 | 0.000% | 0.31 | | ReMILP (Ours) | 0.87s | 87 | 0.077% | 1.22 | | FORGE† | 0.57s | 13 | 10.173% | 102.10 | | Supervised | 0.52s | 16 | 12.103% | 121.38 OTS-hard | Gurobi Default | 67.20s | 731 | 0.000% | 48.38 | | ReMILP (Ours) | 58.31s | 719 | 0.000% | 39.89 | | FORGE† | 51.56s | 648 | 1.762% | 52.96 | | Supervised | 49.21s | 667 | 7.010% | 106.96 CFLP-medium | Gurobi Default | 7.97s | 942 | 0.000% | 0.34 | | ReMILP (Ours) | 4.93s | 626 | 0.029% | 0.52 | | FORGE† | 4.08s | 551 | 0.030% | 0.54 | | Supervised | 3.73s | 513 | 0.031% | 0.54 LB-hard | Gurobi Default | 900.68s | 15,020 | 0.006% | 5.59 | | ReMILP (Ours) | 909.20s | 13,777 | 0.003% | 7.06 | | FORGE† | 879.61s | 13,796 | 0.000% | 7.06 | | Supervised | 862.92s | 15,190 | 0.000% | 6.12

For this experiment, we used Gurobi 12.0 single-threaded on a Ryzen 9 9950X CPU, with a 1000s time limit per instance. Since all three models' highest-confidence predictions tend to agree, we use less conservative values for the settings (k0,k1,Δ)(k_0, k_1, \Delta) of P&S to better distinguish their effects on solver performance.. We set Δ=2\Delta = 2 for all four classes, (k0,k1)=(90(k_0, k_1) = (90\\%, 2\\%) for MMCN-medium-BC, (k0,k1)=(45(k_0, k_1) = (45\\%, 1\\%) for OTS-hard, (k0,k1)=(75(k_0, k_1) = (75\\%, 8\\%) for CFLP-medium, and (k0,k1)=(20(k_0, k_1) = (20\\%, 45\\%) for LB-hard. These settings were chosen based on the distribution of binary variable assignments in the training instances' solution pools.

W2 & Q2: FORGE Comparison

We addressed the concern regarding our FORGE baseline by evaluating the official pretrained FORGE checkpoint released by Shafi et al. on Hugging Face.

  1. Official FORGE Checkpoint: We evaluated the official released checkpoint under both frozen and fine-tuned regimes across all 46 node-level tasks (relative geomean ↓\downarrow) and integrality gap prediction (MAE ↓\downarrow). In both settings, the official model performs substantially worse than our FORGE† reimplementation and is consistently worse than our randomly initialized GNN baseline. Pretraining: ReMILP/FORGE† on MILP-Evolve, official FORGE on MIPLIB+D-MIPLIB. Evaluation: solution/activity on D-MIPLIB, gap on MILP-Evolve.

Frozen encoders

Task GroupRandom (Frozen)FORGE† (Frozen)Official FORGE (Frozen)ReMILP (Frozen)
Binary solution, ILP1.0000.918 ±\pm 0.0071.1460.887 ±\pm 0.016
Binary solution, MILP1.0000.755 ±\pm 0.0121.3670.688 ±\pm 0.050
Constraint activity, ILP1.0000.882 ±\pm 0.0181.2220.871 ±\pm 0.007
Constraint activity, MILP1.0000.863 ±\pm 0.0241.8310.855 ±\pm 0.028
Integrality gap0.0634 ±\pm 0.00840.0530 ±\pm 0.00660.21890.0398 ±\pm 0.0019

Fine-tuned encoders

Task GroupSupervisedFORGE†Official FORGEReMILP
Binary solution, ILP1.0000.994 ±\pm 0.0041.2561.001 ±\pm 0.009
Binary solution, MILP1.0000.919 ±\pm 0.0651.4540.850 ±\pm 0.024
Constraint activity, ILP1.0000.999 ±\pm 0.0041.3400.994 ±\pm 0.004
Constraint activity, MILP1.0000.983 ±\pm 0.0301.7410.959 ±\pm 0.018
Integrality gap0.0426 ±\pm 0.00370.0468 ±\pm 0.00550.23440.0366 ±\pm 0.0014
  1. FORGE sensitivity: FORGE relies on vector-quantized autoencoding rather than contrastive negatives, so negative count does not apply in this case. As for pretraining compute, FORGE† reaches its best validation loss by step 6000, so pretraining for longer doesn't help.
AuthorsOpenReview ↗

Substitution family and soft equivariance

W3 & Q3: Richness of the Substitution Family

We appreciate the opportunity to clarify the reasons behind some of our design choices:

Number of Partners:

There are two reasons for which we do not consider variants of ReMILP that use more than one partners. The first reason is computational. Assuming we implement a hypernetwork that takes in a kk-partner substitution. Such a hypernetwork would require (4+kh)⋅h2(4+ kh) \cdot h^2 parameters where hh is our node embedding dimension. In our current setup, with h=128h = 128 and k=1k=1, the hypernetwork already contains 3.29 times as many parameters as our GNN backbone. Increasing kk further would be wasteful as the hypernetwork is discarded after pretraining. The second reason is mathematical: using rank one substitutions doesn't restrict the generality of Theorems 1 & 2. Indeed, these elementary transformations generate the full group of valid substitutions, so restricting our augmentation distribution to them does not limit the algebraic family of transformations accessible through composition. While approximate equivariance learned on individual generators does not guarantee equivariance to arbitrary compositions, it provides a principled and computationally tractable training objective.

λ\lambda Range:

While we have too many hyperparameters to ablate each one individually, we analyzed the 54 completed trials of our hyperparameter tuning sweep (2 of the 56 had crashed), where λbound\lambda_{\text{bound}} was explored on a log scale over [1.5,6.0][1.5, 6.0]. Across all the sweep trials, the mean validation objective varies by less than 0.0070.007 (0.78930.7893 to 0.79590.7959, with overall IQR 0.00860.0086), showing low sensitivity across the range. The best trial has λbound=4.01\lambda_{\text{bound}} = 4.01, and the top 5 trials also have λbound∈[4.0,5.9]\lambda_{\text{bound}} \in [4.0, 5.9], but that could be due to Optuna's TPE sampler finding a good value of λbound\lambda_{\text{bound}} and holding it constant while tuning the other hyperparameters. Regardless, the exact choice of λbound\lambda_{\text{bound}} doesn't seem to have a big impact given how little the optuna objective changed over the course of the sweep.

λbound\lambda_{\text{bound}} RangeTrial CountObjective Mean (↓\downarrow)Min (Best)Max (Worst)
[1.5,2.0)[1.5, 2.0)70.7922±0.00410.7922\pm 0.00410.78590.7988
[2.0,3.0)[2.0, 3.0)60.7959±0.00910.7959\pm 0.00910.78710.8116
[3.0,4.0)[3.0, 4.0)90.7918±0.00590.7918 \pm 0.00590.78360.7999
[4.0,6.0][4.0, 6.0]320.7893±0.01080.7893\pm 0.01080.77430.8375

Share of Integer Transvections:

For the conditions of theorem 1 to hold, we require integer variables to only undergo integer transvections, and allow continuous variables to undergo any real transvection. As such, the share of integer transvections applied to a given instance is equal to the share of integer variables in that instance.

W4: Soft Equivariance

We agree that our equivariance is learned rather than architecturally guaranteed: the hypernetwork is not constrained to define a group representation. Our claim is therefore empirical approximate equivariance, supported by action-conditioned retrieval on held-out classes, rather than exact equivariance under arbitrary compositions.

AuthorsOpenReview ↗

Transfer to unseen instances, partner embedding, and re-description pretraining cost

W5: Transfer to Unseen Classes

We acknowledge that our previous results do not cleanly establish out-of-distribution transfer. Indeed, the test set for integrality gap predicition comes from the same dataset (MILP Evolve) as that used for pretraining, and the test set for solution and activity prediction comes from the same dataset (D-MIPLIB) as that used for hyperparameter tuning.

As such, we added three new problem families excluded entirely from encoder pretraining and pretraining hyperparameter selection: araujo2015 (Multi-Item Lot Sizing), bollapragada2001 (Structural Member Sizing) and zetina2019 (Network Design). All three were made using the generators provided by FrontierOR [1] and solved using Gurobi like the other classes. We evaluate retrieval, as well as frozen and fine-tuned solution prediction on these three classes, and the results confirm that our representations transfer cleanly to unseen classes.

Invariance and equivariance evaluation

These three real-world families do not exhibit the symmetry present in some of D-MIPLIB's synthetic problem classes. As such, they provide even stronger evidence that our representations carry the inductive biases we intended, with an equivariance Top-1 accuracy of 43.943.9\\% and an equivariance MRR of 53.253.2\\%.

Model | Invariance Top-1 (↑\uparrow) | Invariance MRR (↑\uparrow) | Equivariance Top-1 (↑\uparrow) | Equivariance MRR (↑\uparrow)

  • | - | - | - | - Random | 0.116 ±\pm 0.010 | 0.180 ±\pm 0.011 | 0.033 ±\pm 0.008 | 0.048 ±\pm 0.007 FORGE† | 0.130 ±\pm 0.033 | 0.196 ±\pm 0.036 | 0.030 ±\pm 0.003 | 0.041 ±\pm 0.003 Official FORGE | 0.146 ±\pm 0.001 | 0.192 ±\pm 0.000 | 0.003 ±\pm 0.000 | 0.014 ±\pm 0.000 ReMILP | 0.374 ±\pm 0.052 | 0.459 ±\pm 0.052 | 0.050 ±\pm 0.007 | 0.063 ±\pm 0.007 ReMILP + hyper | - | - | 0.439 ±\pm 0.019 | 0.532 ±\pm 0.017

Frozen encoders

Training a head for solution prediction on frozen ReMILP representations over 50k steps leads to an 8.8% improvement over training one on a randomly initialized GNN on average. FORGE† leads to a 0.9% improvement over Random, and the official FORGE checkpoint's representations lead to a 20% worse KL than our randomly initialized GNN.

Problem class | Random (Frozen) | FORGE† (Frozen) | Official FORGE (Frozen) | ReMILP (Ours, Frozen)

  • | - | - | - | - araujo2015 | 0.6418 ±\pm 0.0050 | 0.6329 ±\pm 0.0210 | 0.6865 ±\pm 0.0000 | 0.5143 ±\pm 0.0056 bollapragada2001 | 0.2537 ±\pm 0.0046 | 0.2503 ±\pm 0.0015 | 0.3005 ±\pm 0.0000 | 0.2403 ±\pm 0.0041 zetina2019 | 0.2639 ±\pm 0.0000 | 0.2639 ±\pm 0.0001 | 0.3599 ±\pm 0.0000 | 0.2633 ±\pm 0.0002

Fine-tuned encoders

Once we fine-tune each of our models for 50k steps on these 3 classes, we observe that ReMILP retains a 4.74.7\\% improvement over a model that is supervised from scratch, compared to 2.32.3\\% for FORGE†. Fine-tuned official FORGE has, on average, 15.1% higher KL than our supervised baseline, further suggesting that its architecture may limit performance on these tasks.

Problem ClassSupervisedFORGE†Official FORGEReMILP
araujo20150.4682 ±\pm 0.01820.4762 ±\pm 0.00940.6698 ±\pm 0.00040.4640 ±\pm 0.0013
bollapragada20010.2222 ±\pm 0.00030.2337 ±\pm 0.00920.2366 ±\pm 0.00070.2237 ±\pm 0.0005
zetina20190.2640 ±\pm 0.00010.2301 ±\pm 0.00140.2641 ±\pm 0.00010.2291 ±\pm 0.0016

[1] Kong, Minwei, et al. "FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization.", NeurIPS 2026 Evaluation & Dataset Track

Q4: Removing the Partner Embedding from the Hypernetwork

Using these same 3 classes once again to test equivariance without partner embeddings, we observe a 12.412.4 point degradation of equivariance retrieval accuracy.

ModelEquivariance Top-1 (↑\uparrow)Equivariance MRR (↑\uparrow)
Without Partner Embedding0.036 ±\pm 0.0040.047 ±\pm 0.004
Full ReMILP0.050 ±\pm 0.0070.063 ±\pm 0.007
Without Partner Embedding + Hypernetwork0.315 ±\pm 0.0150.418 ±\pm 0.019
Full ReMILP + Hypernetwork0.439 ±\pm 0.0190.532 ±\pm 0.017

Q5: Pretraining Time Cost of Constraint Re-Descriptions

We benchmarked exact wall-clock pretraining times on an NVIDIA RTX 5090 across 20,000 steps:

  • 0% redundant constraints: 62.3 min.
  • 5% redundant constraints: 67.0 min (+7.5% overhead).
  • 10% redundant constraints: 69.6 min (+11.7% overhead).
  • 20% redundant constraints: 74.7 min (+19.9% overhead).

Thus, the pretraining time increases linearly in the percentage of added redundant constraints, which is to be expected given both the forward and backward passes of our GNN encoder have a time complexity that is linear in the size of the input.

AuthorsOpenReview ↗

Revised paper

Dear Reviewer UEpq,

We have uploaded a revised version of our manuscript (with changes highlighted in blue) integrating the updates from our rebuttal, including the Gurobi Predict-and-Search evaluation (Section 5.2.4), the official FORGE checkpoint benchmarks (Appendix B) and the out-of-distribution transfer experiments (Appendix C).

As the discussion period is coming to an end in two days, we wanted to make sure you saw the revised paper and check if you have had a chance to go through our responses. Please let us know if your concerns have been addressed. We would be happy to clarify any remaining points or answer further questions.

Thank you very much!

Comments on the paper

1 comment

Dear Reviewer K9Su,

Thank you for your follow-up. We would like to clarify that ReMILP does not use an EMA or momentum teacher. The same encoder processes both reformulated MILPs during pretraining. Variable embeddings are encouraged to remain invariant to constraint re-descriptions, while a hypernetwork predicts their changes under variable substitutions. The encoder and hypernetwork are trained jointly with a contrastive loss.

Regarding computational overhead, in our experimental setup, we measured the following total pretraining times across three seeds on a single RTX 5090:

MethodTotal pretraining time
FORGE†39.2 min
ReMILP, ρ=0%\rho = 0\%62.3 min
ReMILP, ρ=5%\rho = 5\%67.0 min
ReMILP, ρ=10%\rho = 10\%69.6 min
ReMILP, ρ=20%\rho = 20\%74.7 min

Here, ρ\rho denotes the redundant-constraint ratio. ReMILP takes approximately 1.6-1.9 times the pretraining time of FORGE†, with all three seeds completing in approximately 1-1.25 hours on a single GPU. This additional cost is confined to pretraining: downstream prediction uses the encoder and task-specific head, without the hypernetwork or reformulation sampling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.