acceptodds
Under review as a conference paper at ICLR 2027

Opt-Judger: Fine-Grained Evaluation of LLM-Based Optimization Modeling via Feasibility Verification

Abstract

Large language models are increasingly used to generate optimization models from natural language descriptions, motivating the development of methods to evaluate their correctness. However, existing evaluators still struggle to detect constraint errors in generated models and provide limited feedback for diagnosing them. These limitations hinder accurate assessment of LLMs’ ability to translate problem requirements into optimization models and offer little guidance for improvement. To address these limitations, we propose Opt-Judger, a solver-backed framework for fine-grained evaluation of LLM-based optimization modeling. Opt-Judger evaluates whether the constraints in generated models faithfully capture the problem requirements by checking whether they permit the same decisions as a reference model representing these requirements. It uses feasibility verification to search for a witness point, which represents a decision that is feasible under one model but infeasible under the other. This point reveals constraint violations that indicate mismatches with the problem requirements and supports error localization. We also construct Opt-Judger-Bench using equivalent reformulations and transformations that simulate common modeling errors. Extensive experiments show that Opt-Judger accurately identifies genuine constraint errors across benchmarks and consistently outperforms existing evaluators on challenging cases.

Primary area:
optimization
Keywords:
Large Language Models, Optimization modeling, Operations Research, Model Evaluation, Solver Verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aUJmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is part of a line of work that studies how to use LLMs for the automated formulation of mixed-integer linear programs (MILPs) – that is, methods that translate a natural language description of an optimization problem into a MILP. An important component of this pipeline is verification: given a benchmark’s gold-standard MILP formulation and an AI-generated MILP formulation, how do we determine whether the two MILP formulations represent the same problem? This task is not straightforward because there are many different ways to formulate the same optimization problem (e.g., by adding slack variables, cutting planes, etc.).

This paper proposes a method to determine whether two input formulations represent the same problem. First, it uses an LLM to determine, for each formulation, which decision variables are “auxiliary” (e.g., slack variables) and which are not—i.e., the “business” variables. The business variables must be shared between the two formulations. It then uses various substitution and elimination techniques to project the two formulations onto the business variables. Finally, it defines another pair of integer programs to determine whether the two formulations’ projected feasible regions are equal. The experiments show that the method has 100% accuracy on all tested reformulations, beating the most similar baseline EquivaMap (Zhai et al., ’25) on transformations that change the MILP’s feasible region but not its optimal solution.

Critical strengths
  • I think it’s conceivable that follow-up work would build on this approach. The research community and industry have paid fair attention to automated MILP formulation in recent years. The most common techniques for evaluating model equivalence, such as solving both models and comparing their optimal objective values, have clear, straightforward failure cases. Better evaluation methods, such as Opt-Judger, are needed.
  • This paper provides a decent conceptual advancement over EquivaMap. For large MILPs, one may not be able to solve to optimality, so it seems important to ensure that, if two MILPs are meant to be reformulations of each other, nearly optimal solutions to one don’t correspond to infeasible solutions to the other.
  • The paper provides a new benchmark dataset in addition to its new method.
Critical weaknesses
  • It would be helpful to provide some limitations of the method. For example, are there types of reformulations that wouldn’t be captured by this method? Are there examples of MILP reformulations that don’t have the same set of business variables?
  • It would be helpful to clarify what the technical advances are in this paper, or if the advances are largely conceptual. Were there any technical challenges? Or are the techniques in Sections 4.2 or 4.4 largely straightforward from prior work?
  • The writing is fine, and the paper was largely easy to read. I just have a few suggestions:
    • The introduction could use a second pass. For example, the sentence “Our insight is that constraints determine which decisions a model permits” is not really pointing out an insight---it’s just stating the definition of a constraint.
    • The term “business variable” was used several times before it was defined, and I didn’t know what it meant.
Questions for authors

Please see my questions in the "Critical Weaknesses" section.

1 comment
Area_Chair_A3drOpenReview ↗

Reminder: Please Engage in the Rebuttal Discussion

Dear Reviewers,

The rebuttal period is already halfway through, and the authors have submitted their responses. We kindly ask you to review the responses and engage in the discussion with the authors as soon as possible.

Thank you for your time and contribution.

Best regards, Area Chair

Reviewer_scePOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper propose Opt-Judger, a solver-backed framework for fine-grained evaluation/diagnostics of LLM-based optimization modeling. Specially, they can identify constraint errors in generated math models. To achieve this, they first align the generated and reference math models in a shared space of problem decisions, and then check feasible-region equivalence by bidirectional feasibility verification. Besides, they also construct Opt-Judger-Bench using equivalent reformulations and controlled transformations to simulate common constraint errors in LLM-based optimization modeling.

Critical strengths
  1. This paper is very well-written, and thus easy to follow.

  2. They study an important but under-explored topic in LLM-based optimization modeling, that is, the evaluation of optimization modeling beyond objective value matching.

Critical weaknesses
  1. [Limitation of Detection Scope]: The proposed method can only identify constraint errors, but cannot deal with optimation modeling errors in decision variables, objective functions, and solver code. This design largely limits the usefulness of the proposed method.

  2. [The Setting Limits the Usability]: The proposed method takes a natural-language problem description, a reference model, and a generated model as input. However, under most real-world optimization modeling cases, the reference model representing the problem requirements is not available. Why would the users still need to do optimization modeling if they already have the reference math model? This also explains why the authors only evaluated representative modeling methods on NLP4LP (not other common NL-to-Opt benchmarks), which provides reference models, using Opt-Judger.

  3. [Hard to Scale]: The authors admit that "In practice, valid reformulations may use different variable names, auxiliary variables, or algebraic forms, causing their feasible regions to be defined over different variable spaces. Opt-Judger therefore constructs the shared business-variable space through variable alignment and projection." However, it would be very difficult, if possible, to construct a shared business-variable space for an optimization problem with millions of decision variables. From Table 6, we can see that the Avg. Variables of Opt-Judger-Bench is 44, which is relatively very small compared to industrial problems. The authors are suggested to study the scalability/performance of their method on larger datasets.

  4. [More Discussion on Projection Failures]: The authors indicated that "If an integer auxiliary is globally unbounded or cannot be locally eliminated, Opt-Judger reports a projection failure. This step only affects variables outside the business-variable space;" Is this the only cases where a projection failure will occur? Will there be other projection failure cases? What are the consequences (empirical studies should be given)?

  5. [Very Trivial Proposition]: Proposition 1 is very trivial. I did not know why the authors put it as a proposition here formally?

  6. [Add A Stronger Baseline]: From Table 1, we can see that Opt-Judger outperforms the baseline EquivaMap only on the sub-category "Latent Misalignment Transformations", in which another baseline WL-test achieves 100% accuracy. What if we add a stronger baseline by ensembling EquivaMap + WL-test (namely either one detector raises non-equivalence, we say it is not equivalent). As a result, this emsembled baseline can achieve 100% accuracy. What will the advantages be for your method then?

Questions for authors

Please check the weak points.

1 comment
Area_Chair_A3drOpenReview ↗

Reminder: Please Engage in the Rebuttal Discussion

Dear Reviewers,

The rebuttal period is already halfway through, and the authors have submitted their responses. We kindly ask you to review the responses and engage in the discussion with the authors as soon as possible.

Thank you for your time and contribution.

Best regards, Area Chair

Reviewer_qV9ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how LLMs might be used to generate solutions to optimizations problems from natural language descriptions. Their finding is that while existing benchmarks properly assess whether the LLM-generated solution can produce the correct optimal value, diagnosing errors still remains difficult. They propose a new framework which checks not only whether the model can obtain the optimal value, but also whether the constraints generated by the model have the same feasible region as a reference solution. To do this, they employ feasibility verification to locate points which may be feasible under one solution but infeasible under another. Furthermore, they construct a benchmark which highlights the gap between solutions that have the correct feasible regions, and solutions that merely attain the optimal value. This work potentially raises new questions on how to improve coverage for verification of LLM-generated optimization solutions.

Critical strengths

The paper provides an interesting approach to verification for LLM-generated solutions. The authors propose projecting feasible regions into a shared decision-variable space, which makes it simpler to perform the verification rather than exhaustively comparing all feasible points.

Critical weaknesses

I think the practical consequences of the additional detected errors could use further discussion. Table 2 shows that generated formulations can recover the correct optimal value, even if there is a feasible-region mismatch. But I am not sure what to make of this. Does this mean that benchmarks like NLP4LP are insufficiently sensitive to the constraints, so these optimization tasks must be designed to be more difficult where the feasible regions must match in order to find the optimal value? Or does this suggest a bias in the model capabilities that cause it to produce incorrect feasible regions? It would be useful to understand whether feasible region mismatch can produce incorrect optimal values more often, for example if the objective changes or the feasible plan is altered.

The models used in Table 2 are not fully specified. What kinds of models are used for prompt engineering/finetuning? Are different scales used?

Questions for authors

In addition to the above, do the feasible region errors increase if you increase the problem scale? For example, Gao et al (2026) [1] find that the transcription of assigning which variables to values increase as the size of problem instance grows. It would be useful to investigate whether Opt-Judger also detects an increasing rate of feasible-region mismatches as the number of variables, constraints, or numerical parameters increases.

[1] Zhiqi Gao, Albert Ge, Alexander Berenbeim, Nathaniel D. Bastian, and Frederic Sala. Models Can Model, But Can’t Bind: Structured Grounding in Text-to-Optimization. Conference on Language Modeling (COLM), 2026. https://arxiv.org/abs/2605.21751

2 comments
Area_Chair_A3drOpenReview ↗

Reminder: Please Engage in the Rebuttal Discussion

Dear Reviewers,

The rebuttal period is already halfway through, and the authors have submitted their responses. We kindly ask you to review the responses and engage in the discussion with the authors as soon as possible.

Thank you for your time and contribution.

Best regards, Area Chair

no additional concerns, it would be interesting to see some more discussion on why larger problem instances may have more hidden constraint errors. I am happy to keep my positive score of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.