- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work claims that use of today's LLMs to formulate optimization problems does not lead to the most efficient formulations, even if they are correct. The authors support their claim by first generating a dataset of mixed-integer linear programming (MILP) optimization problems using an Agentic AI approach that evolves a base set of problems from existing benchmarks into 500+ tasks that warrant the application of optimization modeling techniques. Then, they benchmark current LLMs to show that direct formulation of the problem can sometimes be orders of magnitude less efficient than an 'expert' formulation.
Critical strengths
- I believe that the problem identified by the authors is a sound one.
- The benchmark of 500 tasks has broad coverage of domains and modeling techniques.
Critical weaknesses
- The evaluation feels circular: the benchmark builds in the authors' claim, and then the benchmarking then shows the claim.
- The results would be more convincing if a naturalistic dataset of optimization problems was used.
- My understanding is that only base models were evaluated. The results would be stronger if baselines were stronger, for example remind the LLMs to avoid beginner modeling mistakes, to think about the problem structure, benchmarking of related works, and so on. Note that some of the works mentioned as concurrent are no longer concurrent; authors should update the references to the published versions and consider benchmarking them or discussing why they are not comparable.
Questions for authors
Please address or clarify the critical weaknesses.