acceptodds
Under review as a conference paper at ICLR 2027

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling

Abstract

Optimization modeling formulates real-world decision problems as mathematical programs that solvers can use to find optimal decisions. Large language models (LLMs) can automate this process, but the resulting correct formulations can require substantial time and memory to construct and solve, limiting practical scalability. Therefore, we systematically investigate whether LLMs can identify problem structure from natural-language descriptions and apply suitable optimization modeling techniques to generate mathematical models and solver code that solve the problems correctly and efficiently. To this end, we first curate OptTips, a knowledge base of 50 expert modeling techniques in eight families. Using this knowledge, we develop OptDachshund, a multi-agent framework that transforms problems from existing optimization benchmarks into new tasks for evaluating LLMs' use of modeling techniques. It constructs conventional and expert mathematical models with solver code for the same task and data, providing baselines for correctness and computational cost. The resulting EfficientOpt benchmark contains 561 expert-reviewed tasks with paired reference implementations. Evaluation of 11 representative LLMs reveals an efficiency gap on correctly solved tasks with comparable measurements: for every LLM, most generated programs take longer to solve than their expert counterparts. Within the comparable reference-size subset, 57% of programs with correct objective values and fewer variables and linear constraints have longer recorded solver times. Case studies show that different modeling techniques can achieve the same optimal value at similar recorded cost. Faster solving may not reduce execution time if the code takes longer to prepare data and build the model. LLM optimization modeling should therefore be evaluated for both correctness and computational efficiency.

Primary area:
optimization
Keywords:
Optimization Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZtryOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work claims that use of today's LLMs to formulate optimization problems does not lead to the most efficient formulations, even if they are correct. The authors support their claim by first generating a dataset of mixed-integer linear programming (MILP) optimization problems using an Agentic AI approach that evolves a base set of problems from existing benchmarks into 500+ tasks that warrant the application of optimization modeling techniques. Then, they benchmark current LLMs to show that direct formulation of the problem can sometimes be orders of magnitude less efficient than an 'expert' formulation.

Critical strengths
  1. I believe that the problem identified by the authors is a sound one.
  2. The benchmark of 500 tasks has broad coverage of domains and modeling techniques.
Critical weaknesses
  1. The evaluation feels circular: the benchmark builds in the authors' claim, and then the benchmarking then shows the claim.
  2. The results would be more convincing if a naturalistic dataset of optimization problems was used.
  3. My understanding is that only base models were evaluated. The results would be stronger if baselines were stronger, for example remind the LLMs to avoid beginner modeling mistakes, to think about the problem structure, benchmarking of related works, and so on. Note that some of the works mentioned as concurrent are no longer concurrent; authors should update the references to the published versions and consider benchmarking them or discussing why they are not comparable.
Questions for authors

Please address or clarify the critical weaknesses.

Reviewer_3xzHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores whether LLMs can find efficient and correct formulations and solver code for optimization problems extracted from natural language. It is made up of three things (1) OptTips, a knowledge base modeling techniques (2) OptDachshund, a agentic framework that transform existing optimization benchmarks into new tasks for evaluating LMs for modeling optimization problems (3) EfficientOpt, a benchmark with hundreds of expert-reviewed tasks and reference implementations. They find that correct LM solutions to the optimization problems are still around 1.5-2x slower than expert formulations, while being faster than ordinary formulations. The main novelty is that they make tasks with an explicit label for an efficient solution and an accompanying ordinary and expert formulation.

Critical strengths

The paper introduces a technique for making tasks for evaluating LMs for formulating optimization problems by curating techniques for faster optimization and then making tasks conditioned on those optimization techniques. They create two formulations for each problem, one ordinary and one expert. These tasks can potentially be used to illuminate the strengths and weaknesses of LM-based optimization modeling systems.

Critical weaknesses

This paper should be rejected on the grounds that its novelty is not clear. Existing papers have already explored if language models are able to find fast formulations, such as SAGE, and they have explored how giving the model access to a cache of existing optimization techniques can impact it. This means that the current paper's claimed contribution will mostly be the system that allows formulating new problems with a specific optimization technique. Existing papers have already used LM pipelines for synthesizing new tasks (e.g., OptMATH and Step-Opt) and there are no ablations against humans or single-agent reformulations.

The list of 50 optimization techniques are based on already established techniques that exist in textbooks and papers, and existing papers have already used opitimization technique libraries, such as AlphaOPT, which introduces a library of techniques aimed at correctness. This paper does no comparisons of how well the LMs can formulate the optimization problems if given access to the optimization techniques (OptTips).

Questions for authors

The main area that will make the contributions more convincing there are ablations showing that the OptDachshund system works better than existing approaches, and proof that conditioning the generation of new benchmarks on the 50 techniques actually is useful. Right now, numbers are mostly reported aggregated, rather than per technique (it also seems that only 29/50 techniques were evaluated), when it comes to cost and speedup. The main novelty claim of the paper is that the benchmark has labels with techniques that are paired with ordinary and expert formulations for each problem, it would make the paper much stronger if they look at the per-task speedup when the LMs find the techniques and when they don't. It would be great if the authors could either add this table or explain why it isn't relevant!

Reviewer_dTGuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EfficientOpt to study the cost of LLM-generated optimization programs. OptTips organizes 50 modeling techniques, and OptDachshund builds tasks with ordinary and expert reference implementations. The main evaluation covers 561 tasks in 29 technique groups and 11 general-purpose LLMs. It compares numerical accuracy, technique use, construction time, and solver costs. The results show that matching an optimal objective or using a smaller formulation does not ensure an efficient implementation.

Critical strengths
  1. Paired ordinary and expert references distinguish improving on a conventional implementation from approaching an expert implementation on the same task and data.
  2. The analysis separates objective agreement, technique labels, and cost. Reporting Build+opt and solver Work adds useful information beyond solver runtime.
  3. Shared-task comparisons and checks that exclude small timing differences strengthen the main findings. The benchmark construction also includes independent expert review.
Critical weaknesses
  1. The correctness check may accept incomplete solutions. Section 4.1 checks solver status and objective values, while Definition 1 also requires the decision outputs specified by O\mathcal{O}. Table 14 notes this limit, but the paper does not measure how often programs that pass this check return complete, valid solutions.
  2. The code interface may make iterative methods harder to use and compare fairly. Appendix B.3 asks LLMs to return a Gurobi model, while expert code can coordinate several solves (Appendix H). This matters for Benders decomposition and column generation. Solver calls inside model-building code may also be missing from Runtime and Work, although their time can enter Build+opt (Appendix D.1).
  3. The reliability of the technique labels remains unclear. One LLM judge assigns labels using code, references, and execution results. The three illustrative manual checks in Appendix F.3 do not establish agreement with experts across the benchmark, leaving the reported 46.3% target-technique rate uncertain.
Questions for authors
  1. On a representative sample, how often do programs with matching objective values return all requested decisions and satisfy the original constraints? Does the LLM–expert cost gap remain when decision recovery is timed?
  2. Could the authors run a Benders or column-generation reference through the LLM interface and measure the total time and solver work across all calls? Do the cost findings hold on tasks where both sides use a single solver call?
  3. How well do the labels agree with independent experts on a sample covering different techniques and label classes? Does hiding runtime information from the LLM judge change the labels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.