- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method to evaluate the performance of LLM-based optimization modeling. Their method is inspired by software test: rather than only evaluating the original optimization problem, they propose to augment the test as follows. By treating each target inequality constraint in the optimization formulation as a coverage target, they first generates admissible LP instances by changing this constraint, and then translate back to corresponding NL problem description. By doing so, they can generate a suite of test instances to evaluate each given original natural-language problem and its reference LP.
Critical strengths
-
This paper is very well presented, well-motivated, well-organised, and thus very easy to follow;
-
The authors borrow the idea from software test to LLM-based optimization modeling. This is somewhat new to me, and I learned something useful.
Critical weaknesses
-
[Limitation of Scope]: 1) They can only handle LP models, which leaving other types of optimization models (such as IP, MILP, Nonlinear models, SOCP, etc.) not discussed; 2) This method can only mitigate constraints errors. How to handle/mitigate other types of errors regarding decision variables, objectives, solver codes?
-
[Hard to Scale]: How to handle large scale instances (say each instance has up to millions of constraints as in MIPLIB-NL: https://arxiv.org/abs/2602.10450)?
-
[Unclear Contribution]: The authors point out that Zadorojniy et al. (2025) generate problem-level tests or model-specific mutations to evaluate optimization modeling formulations. However, the pros and cons compared with this existing study is not clear (what are the differences? Why and when people use your method rather than theirs?).
-
[Quality Control is Not Clear Enough]: The authors indicate that "Parameter Mapping and Instance Reconstruction use LLM assistance to establish parameter-to-coefficient mappings and to reconstruct generated parameter settings as complete natural-language problems, respectively." How human experts were involved in this process (How many experts? How much time each person? Etc.)? How to make sure that these two steps were correct ? How to measure the quality of these two steps (What if the LLMs make mistakes?)? Is it possible to totally automate these two steps?
-
[Limitations of Used Benchmarks]: The authors indicate that "Since our method targets linear programs, we filter each benchmark to retain instances whose reference formulations are linear programs with explicit inequality constraints. This yields 48 instances from NLP4LP, 17 from In- dustryOR, 21 from LogiOR, 70 from MAMO (including both EasyLP and ComplexLP), and 1,844 from ReSocratic." and "We focus on formulations that are correct on the original instances." As a result, the number of evaluated instances for each (or some of these models) model may be insufficient (Table 1 gave these numbers for evaluated models) and the obtained conclusions may not be generalizable . As an alternative, why not constructing a tailored/additional benchmark for this test by the authors themselves?
[Minor Issues]: Please use \citep{} for references such as "whereas fine-tuning-based methods adapt LLMs using optimization-specific training data and modeling-oriented supervision Jiang et al. (2025); Huang et al. (2025a); Chen et al. (2026)."
Questions for authors
Please check the weak points.
2 comments
[Unclear Contribution]: The authors point out that Zadorojniy et al. (2025) generate problem-level tests or model-specific mutations to evaluate optimization modeling formulations ...
The main distinction lies in the test suites: their suites consist of executable unit tests that check business requirements, whereas ours consist of multiple numerical instances of the same problem, with explicit coverage targets for individual reference constraints. This enables a more fine-grained assessment of test adequacy at the constraint level. Zadorojniy et al. [1] use LLMs to generate executable unit tests from natural language problem descriptions and a business interface. These tests use assertions to check whether model solutions satisfy the corresponding business requirements. However, passing these assertions does not show that constraints required by the problem have been adequately tested. Their method lacks an overall measure of coverage across individual constraints and a mechanism for systematically generating additional tests based on gaps in constraint coverage. Our work addresses this gap by using constraint coverage as an evaluation metric. Our test suites consist of multiple numerical instances of the same problem. We can quantify how many target constraints a suite covers, identify coverage gaps, and use these gaps to guide the generation of additional instances.
The two approaches suit different settings. When no trusted reference LP is available, their method offers a way to construct tests directly from natural language requirements, and it generates an auxiliary optimization model within its workflow. When a trusted reference LP is available, our method can generate multiple test instances of the same problem to verify whether most constraints in the optimization model are modeled correctly.
[Quality Control is Not Clear Enough]: How human experts were involved in this process ...
In Stage (1), LLM infers the semantic types and admissible ranges of parameters from the problem description, such as positive numbers, nonnegative numbers, positive integers, and proportions within . In Stage (3), we prompt the LLM to reconstruct the problem description using the generated coefficients and established mappings.
Using LLMs to obtain information about parameter values from natural language has precedents in prior work. For example, GPTuner [1] uses LLMs to transform natural-language tuning knowledge from sources such as database documentation into structured information, including recommended values and suggested lower and upper bounds, which is then used to refine the optimization search space. This provides a related precedent for our use of LLMs to assist in determining parameter types and admissible ranges from problem descriptions.
To check output quality in our study, three graduate students manually reviewed the parameter ranges inferred by the LLM. No errors were found in the reviewed outputs. We also used GPT-5-mini to check the reconstructed problem descriptions, and this automated check detected no errors. We will clarify these checking procedures in the Appendix C.8.
[Limitations of Used Benchmarks] The authors indicate that "Since our method targets linear programs ...
Thank you for suggesting the construction of a dedicated benchmark. Currently, there is a lack of optimization modeling datasets dedicated specifically to LP problems. Our current evaluation covers five public benchmarks for automated optimization modeling: NLP4LP, IndustryOR, LogiOR, MAMO, and ReSocratic. From these benchmarks, we retain problems whose reference formulations are continuous LPs with explicit inequality constraints, yielding 48, 17, 21, 70, and 1,844 original problems, respectively, for a total of 2,000. We agree that a dedicated benchmark for constraint coverage testing would help broaden the evaluation and systematically examine different constraint structures and modeling errors. We consider this an important direction for future work.
[Minor Issues]
Thank you for pointing out the citation formatting issue. We have corrected it.
Reference
[1] Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. “GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization.” Proceedings of the VLDB Endowment, 17(8):1939–1952, 2024. doi:10.14778/3659437.3659449.
We sincerely thank you for taking the time to review our paper and for your constructive feedback. We have responded to your concerns as outlined below.
Our current study focuses on LPs because our test generation method uses the KKT optimality conditions of LPs. It searches for parameter settings under which the target constraints take part in these conditions at the optimum, which increases constraint coverage across test instances. Extending this approach to other problem classes, such as IPs, MILPs, nonlinear programs, and SOCPs, would require corresponding changes to the optimality conditions, coverage criteria, and generation procedure. We leave these extensions to future work.
Regarding errors in decision variables, objectives, and solver code, our method uses constraint coverage to guide test generation, but the resulting tests can also expose other types of errors. We generate multiple test instances for each problem and compare the optimal objective values returned by the evaluated and reference models. Thus, errors such as omitted variables or incorrectly specified objectives can be detected when they cause an objective-value mismatch on an additional test instance. Runtime errors can also be identified during test execution. Our focus on hidden constraint errors is also motivated by prior work. Zadorojniy et al. [1] state that "single-constraint mutations are the most challenging to detect." This statement concerns individual constraint mutations rather than a measured comparison across all error categories. Our contribution is to introduce constraint coverage as an explicit testing criterion and to use it to guide the generation of admissible test instances that exercise reference constraints not yet covered. These additional tests help expose modeling errors that remain hidden when the optimal objective value on the original instance matches the reference.
Thank you for your concern about instance size and scalability. We have added the size distributions of the LP problems from the original five datasets (Figure 5) and conducted additional experiments on Bench4Opt. The corresponding statistics and results are reported in Appendix C.7.
For MIPLIB-NL, we examined the problem descriptions, reference implementations, and variable types of each instance, but found no LP problems satisfying our method's requirement. We therefore evaluated our method on 66 continuous LP problems from Bench4Opt, with the largest model containing 280 variables and 68 explicit constraints.
The table below summarizes constraint coverage, instance counts, and generation times on the 66 Bench4Opt LP problems. Times are averaged per generated instance.
These results show that our method extends beyond the original five datasets: on Bench4Opt, it increases constraint coverage by 65.45 percentage points while requiring only 8.60 seconds of Step 2 computation per generated instance on average. The results support the broad applicability of our approach to continuous LP modeling problems with different sources and model sizes. They also show that the coverage improvement extends to the larger models. We identify scalability to problems with millions of constraints as an important direction for future work, contingent on the availability of suitable modeling datasets that pair natural-language problem descriptions with LP models at that scale.
Reference
[1] Alexander Zadorojniy, Segev Wasserkrug, and Eitan Farchi. “An Agent-Based Framework for the Automatic Validation of Mathematical Optimization Models.” arXiv preprint, arXiv:2511.16383, 2025 (revised 2026, v2).