acceptodds
Under review as a conference paper at ICLR 2027

CONSTRAINT COVERAGE GUIDED HIDDEN ERRORS EXPOSURE FOR AUTOMATED OPTIMIZATION MODELLING

Abstract

Optimization modeling translates real-world requirements into mathematical models and executable solver code, and large language models (LLMs) have shown promise in automating this process from natural-language descriptions. However, existing evaluations typically test a generated optimization model only on the original numerical instance and judge correctness by its optimal objective value. From a software-testing perspective, this resembles executing a program on a single input and checking only its final output: constraints irrelevant to the original optimum may remain untested. We introduce a constraint-coverage-guided validation paradigm for automated linear programming modeling, which assesses whether each target inequality constraint contributes to establishing optimality in at least one test instance. Inspired by coverage-guided software testing and inverse optimization, our framework treats constraints as test targets and searches prescribed parameter domains for problem variants that exercise previously uncovered constraints. It then reconstructs the generated variants as complete natural-language problems, producing an augmented test suite of paired problem descriptions and reference models. Experiments on public optimization-modeling benchmarks show that the augmented test suites substantially increase constraint coverage and expose hidden constraint-level modeling errors in optimization models produced by automated optimization modeling systems, even when these formulations yield correct objective values on the original instances. These results demonstrate that constraint coverage provides a more diagnostic complement to conventional answer-based evaluation.

Primary area:
optimization
Keywords:
Optimization Formulation Evaluation, LLM, Data Augmentation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_hihgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method to evaluate the performance of LLM-based optimization modeling. Their method is inspired by software test: rather than only evaluating the original optimization problem, they propose to augment the test as follows. By treating each target inequality constraint in the optimization formulation as a coverage target, they first generates admissible LP instances by changing this constraint, and then translate back to corresponding NL problem description. By doing so, they can generate a suite of test instances to evaluate each given original natural-language problem and its reference LP.

Critical strengths
  1. This paper is very well presented, well-motivated, well-organised, and thus very easy to follow;

  2. The authors borrow the idea from software test to LLM-based optimization modeling. This is somewhat new to me, and I learned something useful.

Critical weaknesses
  1. [Limitation of Scope]: 1) They can only handle LP models, which leaving other types of optimization models (such as IP, MILP, Nonlinear models, SOCP, etc.) not discussed; 2) This method can only mitigate constraints errors. How to handle/mitigate other types of errors regarding decision variables, objectives, solver codes?

  2. [Hard to Scale]: How to handle large scale instances (say each instance has up to millions of constraints as in MIPLIB-NL: https://arxiv.org/abs/2602.10450)?

  3. [Unclear Contribution]: The authors point out that Zadorojniy et al. (2025) generate problem-level tests or model-specific mutations to evaluate optimization modeling formulations. However, the pros and cons compared with this existing study is not clear (what are the differences? Why and when people use your method rather than theirs?).

  4. [Quality Control is Not Clear Enough]: The authors indicate that "Parameter Mapping and Instance Reconstruction use LLM assistance to establish parameter-to-coefficient mappings and to reconstruct generated parameter settings as complete natural-language problems, respectively." How human experts were involved in this process (How many experts? How much time each person? Etc.)? How to make sure that these two steps were correct ? How to measure the quality of these two steps (What if the LLMs make mistakes?)? Is it possible to totally automate these two steps?

  5. [Limitations of Used Benchmarks]: The authors indicate that "Since our method targets linear programs, we filter each benchmark to retain instances whose reference formulations are linear programs with explicit inequality constraints. This yields 48 instances from NLP4LP, 17 from In- dustryOR, 21 from LogiOR, 70 from MAMO (including both EasyLP and ComplexLP), and 1,844 from ReSocratic." and "We focus on formulations that are correct on the original instances." As a result, the number of evaluated instances for each (or some of these models) model may be insufficient (Table 1 gave these numbers for evaluated models) and the obtained conclusions may not be generalizable . As an alternative, why not constructing a tailored/additional benchmark for this test by the authors themselves?

[Minor Issues]: Please use \citep{} for references such as "whereas fine-tuning-based methods adapt LLMs using optimization-specific training data and modeling-oriented supervision Jiang et al. (2025); Huang et al. (2025a); Chen et al. (2026)."

Questions for authors

Please check the weak points.

2 comments

We sincerely thank you for taking the time to review our paper and for your constructive feedback. We have responded to your concerns as outlined below.

[Limitation of Scope] 1) They can only handle LP models ..., 2) This method can only mitigate constraints errors ...

Our current study focuses on LPs because our test generation method uses the KKT optimality conditions of LPs. It searches for parameter settings under which the target constraints take part in these conditions at the optimum, which increases constraint coverage across test instances. Extending this approach to other problem classes, such as IPs, MILPs, nonlinear programs, and SOCPs, would require corresponding changes to the optimality conditions, coverage criteria, and generation procedure. We leave these extensions to future work.

Regarding errors in decision variables, objectives, and solver code, our method uses constraint coverage to guide test generation, but the resulting tests can also expose other types of errors. We generate multiple test instances for each problem and compare the optimal objective values returned by the evaluated and reference models. Thus, errors such as omitted variables or incorrectly specified objectives can be detected when they cause an objective-value mismatch on an additional test instance. Runtime errors can also be identified during test execution. Our focus on hidden constraint errors is also motivated by prior work. Zadorojniy et al. [1] state that "single-constraint mutations are the most challenging to detect." This statement concerns individual constraint mutations rather than a measured comparison across all error categories. Our contribution is to introduce constraint coverage as an explicit testing criterion and to use it to guide the generation of admissible test instances that exercise reference constraints not yet covered. These additional tests help expose modeling errors that remain hidden when the optimal objective value on the original instance matches the reference.

[Hard to Scale] How to handle large scale instances ...

Thank you for your concern about instance size and scalability. We have added the size distributions of the LP problems from the original five datasets (Figure 5) and conducted additional experiments on Bench4Opt. The corresponding statistics and results are reported in Appendix C.7.

For MIPLIB-NL, we examined the problem descriptions, reference implementations, and variable types of each instance, but found no LP problems satisfying our method's requirement. We therefore evaluated our method on 66 continuous LP problems from Bench4Opt, with the largest model containing 280 variables and 68 explicit constraints.

The table below summarizes constraint coverage, instance counts, and generation times on the 66 Bench4Opt LP problems. Times are averaged per generated instance.

MetricOriginalAugmented
Number of instances66760
Constraint coverage (%)30.3095.75
Step 1: mean time per instance (s)---1.69
Step 2: mean time per instance (s)---8.60
Step 3: mean time per instance (s)---49.98

These results show that our method extends beyond the original five datasets: on Bench4Opt, it increases constraint coverage by 65.45 percentage points while requiring only 8.60 seconds of Step 2 computation per generated instance on average. The results support the broad applicability of our approach to continuous LP modeling problems with different sources and model sizes. They also show that the coverage improvement extends to the larger models. We identify scalability to problems with millions of constraints as an important direction for future work, contingent on the availability of suitable modeling datasets that pair natural-language problem descriptions with LP models at that scale.

Reference

[1] Alexander Zadorojniy, Segev Wasserkrug, and Eitan Farchi. “An Agent-Based Framework for the Automatic Validation of Mathematical Optimization Models.” arXiv preprint, arXiv:2511.16383, 2025 (revised 2026, v2).

[Unclear Contribution]: The authors point out that Zadorojniy et al. (2025) generate problem-level tests or model-specific mutations to evaluate optimization modeling formulations ...

The main distinction lies in the test suites: their suites consist of executable unit tests that check business requirements, whereas ours consist of multiple numerical instances of the same problem, with explicit coverage targets for individual reference constraints. This enables a more fine-grained assessment of test adequacy at the constraint level. Zadorojniy et al. [1] use LLMs to generate executable unit tests from natural language problem descriptions and a business interface. These tests use assertions to check whether model solutions satisfy the corresponding business requirements. However, passing these assertions does not show that constraints required by the problem have been adequately tested. Their method lacks an overall measure of coverage across individual constraints and a mechanism for systematically generating additional tests based on gaps in constraint coverage. Our work addresses this gap by using constraint coverage as an evaluation metric. Our test suites consist of multiple numerical instances of the same problem. We can quantify how many target constraints a suite covers, identify coverage gaps, and use these gaps to guide the generation of additional instances.

The two approaches suit different settings. When no trusted reference LP is available, their method offers a way to construct tests directly from natural language requirements, and it generates an auxiliary optimization model within its workflow. When a trusted reference LP is available, our method can generate multiple test instances of the same problem to verify whether most constraints in the optimization model are modeled correctly.

[Quality Control is Not Clear Enough]: How human experts were involved in this process ...

In Stage (1), LLM infers the semantic types and admissible ranges of parameters from the problem description, such as positive numbers, nonnegative numbers, positive integers, and proportions within [0,1][0,1]. In Stage (3), we prompt the LLM to reconstruct the problem description using the generated coefficients and established mappings.

Using LLMs to obtain information about parameter values from natural language has precedents in prior work. For example, GPTuner [1] uses LLMs to transform natural-language tuning knowledge from sources such as database documentation into structured information, including recommended values and suggested lower and upper bounds, which is then used to refine the optimization search space. This provides a related precedent for our use of LLMs to assist in determining parameter types and admissible ranges from problem descriptions.

To check output quality in our study, three graduate students manually reviewed the parameter ranges inferred by the LLM. No errors were found in the reviewed outputs. We also used GPT-5-mini to check the reconstructed problem descriptions, and this automated check detected no errors. We will clarify these checking procedures in the Appendix C.8.

[Limitations of Used Benchmarks] The authors indicate that "Since our method targets linear programs ...

Thank you for suggesting the construction of a dedicated benchmark. Currently, there is a lack of optimization modeling datasets dedicated specifically to LP problems. Our current evaluation covers five public benchmarks for automated optimization modeling: NLP4LP, IndustryOR, LogiOR, MAMO, and ReSocratic. From these benchmarks, we retain problems whose reference formulations are continuous LPs with explicit inequality constraints, yielding 48, 17, 21, 70, and 1,844 original problems, respectively, for a total of 2,000. We agree that a dedicated benchmark for constraint coverage testing would help broaden the evaluation and systematically examine different constraint structures and modeling errors. We consider this an important direction for future work.

[Minor Issues]

Thank you for pointing out the citation formatting issue. We have corrected it.

Reference

[1] Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. “GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization.” Proceedings of the VLDB Endowment, 17(8):1939–1952, 2024. doi:10.14778/3659437.3659449.

Reviewer_CtbyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a constraint-coverage-guided validation framework for automated optimization modeling. Instead of evaluating a generated formulation only on the original numerical instance, the method identifies inactive or linearly dependent active reference constraints and generates additional LP instances in which previously uncovered constraints participate in establishing optimality. The framework combines dynamic parameter tracing, a KKT-based coverage-guided objective search, and natural-language instance reconstruction. Experiments on five optimization-modeling benchmarks show higher constraint coverage and expose hidden constraint-level errors missed by single-instance answer matching.

Critical strengths
  1. The paper studies an important reliability issue in automated optimization modeling: constraint omissions or overly relaxed/misformulated constraints can remain undetected when they do not affect the optimum of the original numerical instance.

  2. The constraint-coverage formulation is interesting and technically well motivated. The method distinguishes inactive and dependent active constraints and uses a KKT-based CGOS problem to search for parameter settings that exercise previously uncovered constraints.

  3. The experiments across multiple benchmarks and modeling systems provide evidence that the augmented test suites improve constraint coverage and can expose hidden constraint-level errors missed by conventional answer-based evaluation.

Critical weaknesses
  1. The related-work discussion should be strengthened. Recent work has already moved beyond objective-value agreement toward constraint- or structure-level verification, including Constraint Injection for VRP modeling [1] and Opt-Verifier [2]. The paper should clarify more explicitly how constraint coverage and optimization-guided instance generation differ from, and complement, these approaches.

  2. The computational cost and success rate of the full constraint-coverage-guided generation pipeline are not reported in enough detail. It would be useful to report the number of CGOS solves and candidate perturbations, per-problem generation time, and the fraction of attempts that actually yield a new coverage witness.

  3. Table 1 reports no newly exposed hidden constraint errors on IndustryOR for any evaluated modeling method. Since constraint coverage still increases substantially on this dataset, the paper should explain why the additional coverage does not translate into newly detected errors.

  4. The current study is restricted to continuous LPs. Results on MILP formulations would substantially strengthen the evidence that the proposed hidden-error detection paradigm applies to practically important discrete optimization settings.

Reference:

[1] Xizi Luo, Changhong He, Dongdong Geng, Chenggong Shi, and Yu Mei. "Beyond Objective Equivalence: Constraint Injection for LLM-Based Optimization Modeling on Vehicle Routing Problems." arXiv preprint arXiv:2606.04816, 2026.

[2] Haoyang Liu, Jie Wang, Boxuan Niu, Xiongwei Han, Yian Xu, Mingxuan Ye, Zijie Geng, Fangzhou Zhu, Tao Zhong, Mingxuan Yuan, and Jianye Hao. "Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification." Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:78808-78849, 2026.

Questions for authors
  1. What is the computational cost of generating the augmented test suite, including the number of CGOS solves/candidate perturbations, wall-clock time, and the success rate of obtaining new coverage witnesses?

  2. Why does IndustryOR show no newly exposed hidden constraint errors in Table 1 despite the substantial increase in constraint coverage?

  3. How does the proposed framework differ from recent constraint- or structure-level verification methods such as Constraint Injection [1] and Opt-Verifier [2]?

  4. Can the authors provide preliminary evidence on extending the coverage criterion and generation procedure to MILP formulations?

Reference:

[1] Xizi Luo, Changhong He, Dongdong Geng, Chenggong Shi, and Yu Mei. "Beyond Objective Equivalence: Constraint Injection for LLM-Based Optimization Modeling on Vehicle Routing Problems." arXiv preprint arXiv:2606.04816, 2026.

[2] Haoyang Liu, Jie Wang, Boxuan Niu, Xiongwei Han, Yian Xu, Mingxuan Ye, Zijie Geng, Fangzhou Zhu, Tao Zhong, Mingxuan Yuan, and Jianye Hao. "Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification." Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:78808-78849, 2026.

2 comments

We are very grateful for your careful review of our manuscript and for your helpful comments. Our responses to your concerns are provided below.

What is the computational cost of generating the augmented test suite, including the number of CGOS solves/candidate perturbations, wall-clock time, and the success rate of obtaining new coverage witnesses?

We have added these statistics for the 2,000 original LP problems across five datasets in Appendix C.6. We summarize the results below. CGOS solve counts and candidate parameter perturbation counts are averaged per newly generated instance. The new-witness rate is the fraction of evaluated attempts that produce a verified optimal witness covering at least one previously uncovered target constraint. Attempts include evaluations of parameter-perturbed candidates and searches that modify only the objective function; unsuccessful attempts are included in the denominator.

DatasetMean CGOS solvesMean parameter perturbationsNew witnesses / attemptsNew-witness rate (%)
NLP4LP3.435.6444/28215.60
IndustryOR4.073.2914/6023.33
LogiOR6.228.7046/42810.75
MAMO0.196.64667/4,50814.80
ReSocratic2.362.742,020/7,07028.57

We also report the mean time for each stage and the LLM costs below. Times are averaged per newly generated instance, while costs are averaged per original problem.

DatasetStep (1) mean time (s)Step (2) mean time (s)Step (3) mean time (s)Step (1) mean cost (USD)Step (3) mean cost (USD)
NLP4LP10.760.13913.890.00200.0028
IndustryOR13.850.26117.500.00320.0037
LogiOR7.990.39519.140.00440.0107
MAMO2.372.11121.390.00630.0558
ReSocratic11.640.21416.940.00310.0049

The amortized Step (2) runtime is below 0.4 seconds per generated instance on four datasets and 2.11 seconds on MAMO, supporting efficient test suite expansion across the evaluated datasets. Across datasets, 10.75%–28.57% of evaluated attempts produce a verified new coverage witness, showing that the search converts candidate exploration into additional constraint coverage. A key feature of our method is that it explicitly directs instance-generation attempts toward currently uncovered target constraints. Consistent with this design, Table 5 in Appendix C.4 shows that, others does not consistently reach the target coverage achieved by our complete method, whose aggregate constraint coverage reaches 95% or higher on all five datasets.

Why does IndustryOR show no newly exposed hidden constraint errors in Table 1 despite the substantial increase in constraint coverage?

Thank you for drawing attention to this result. On IndustryOR, the evaluated formulations that produced correct answers on the original instances continued to agree with the reference answers on the expanded test suites. Consequently, no additional hidden constraint errors were exposed. Our work starts from the observation that agreement on the original instance can conceal constraint modeling errors. Table 1 presents cases in which formulations that pass the original answer-based evaluation behave differently under changed parameter values, revealing differences in modeling quality that the original evaluation could not distinguish. Although no such errors were exposed on IndustryOR, the results on other datasets demonstrate this phenomenon. Our aim is to reveal these differences and provide additional evaluation evidence through constraint coverage guided test expansion. The number of errors exposed on each dataset is one aspect of the experimental results and serves to compare the accuracy of the evaluated optimization modeling methods under the expanded tests. The improvement in constraint coverage shows that the expanded test suites exercised more target constraints that the original instances left uncovered.

How does the proposed framework differ from recent constraint- or structure-level verification methods such as Constraint Injection [1] and Opt-Verifier [2]?

Thank you for pointing out the need to clarify the differences between our framework and these two studies. We have added this discussion to Section 2.2, Related Work.

Comparison with Opt-Verifier.

We have different research objectives. Without requiring an existing ground-truth reference LP, Opt-Verifier uses LLMs during modeling to back-translate an optimization model into an abstract modeling structure, compare it with the structure extracted from the problem description, and refine the model based on feedback. This structural check depends on both the back-translation capabilities of LLMs and their understanding of abstract modeling structures. It provides a verification mechanism beyond objective-value comparison during the modeling process.

Our work approaches the problem from the perspective of test adequacy, investigating how to evaluate the correctness of generated formulations more thoroughly. Agreement between a generated model and a reference model on the original instance does not establish that they remain consistent under other admissible parameter settings. We therefore use reference constraint coverage to guide test-suite augmentation and compare the solutions produced by the same generated formulation with those of the reference model across multiple numerical instances. This reveals differences in modeling quality that a single instance cannot distinguish and provides additional evidence for assessing modeling correctness.

Comparison with Constraint Injection.

Our core generation mechanism applies to continuous LPs across different application domains, whereas the implementation of Constraint Injection focuses on the Vehicle Routing Problem (VRP). That study also recognizes that matching optimal objective values can conceal constraint modeling errors. It uses VRP-specific relationships, such as vehicle capacities, customer time windows, and pickup–delivery precedence, to construct feasible routes and routes violating a particular constraint family through family-specific perturbation operators. These routes are then encoded as additional constraints and injected into candidate models. Solver-based feasibility queries check whether the models correctly accept feasible routes and reject violating routes. This process relies on domain knowledge about VRP constraints and route representations. The authors also explicitly state that transfer to other optimization domains requires corresponding expert-written templates, constraint-family-specific perturbation operators, and compatible solution-encoding rules.

Our framework uses reference constraint coverage as both an evaluation metric and a generation criterion. For continuous LPs arising from different application domains, we uniformly treat target inequalities in the reference model as coverage targets. Using the KKT optimality conditions for LPs, we search admissible parameter domains for instances in which previously uncovered constraints enter a linearly independent working set at an optimal solution. The core generation mechanism thus uses the mathematical structure of LPs, without requiring dedicated route or decision perturbation operators for each business constraint. This enables a unified coverage evaluation and generation mechanism for systematically augmenting test suites across LP application domains.

Can the authors provide preliminary evidence on extending the coverage criterion and generation procedure to MILP formulations?

Thank you for highlighting the importance of extending the method to MILPs to assess its practical applicability. Our current study focuses on continuous LPs because our test generation mechanism uses the KKT optimality conditions for LPs. Within admissible parameter domains, it searches for instances in which previously uncovered constraints enter a linearly independent working set at an optimal solution, thereby increasing constraint coverage. This mechanism relies on the optimality structure of continuous LPs and cannot be directly applied to models containing integer variables.

For MILPs, satisfying the KKT conditions of the continuous relaxation is insufficient to establish optimality for the integer model. Integer feasibility and discrete optimality must also be considered, requiring further investigation of appropriate coverage criteria and instance generation mechanisms. We agree that experiments on MILPs would provide broader evidence for this research direction. This paper first establishes and evaluates constraint coverage guided test-suite augmentation for continuous LPs, and extending it to MILPs is an important direction for future work.

Reviewer_ExubOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an automatic optimization modeling and evaluation framework based on Constraint Coverage, aimed at discovering hidden errors in LLM-generated optimization models that cannot be detected through a single optimal target value comparison. This method combines inverse optimization and coverage-guided testing, generates new test instances that can activate different constraints by adjusting the objective function and constraint parameters, and reconstructs the corresponding natural language problem using an LLM. Experiments on five publicly available datasets have shown that this method increases constraint coverage from 45% -72% to 94% -98%, and successfully identifies modeling errors in constraints that were overlooked by traditional evaluations.

Critical strengths
  1. The paper identifies an important limitation of conventional answer-based evaluation and introduces constraint coverage as a complementary criterion for detecting hidden modeling errors.

  2. The framework effectively integrates parameter mapping, optimization-guided test generation, and natural-language reconstruction into a coherent automated evaluation pipeline.

Critical weaknesses
  1. An active, linearly independent constraint does not necessarily contribute to optimality. The proposed criterion may therefore overestimate meaningful constraint coverage, particularly in degenerate LPs.

  2. Despite substantial coverage improvements, only 59 hidden errors are exposed among 9,601 originally correct formulations (approximately 0.6%). The paper lacks systematic comparisons of error-detection effectiveness against alternative test-generation approaches.

  3. The framework relies on LLM-inferred parameter domains and reconstructed problem descriptions, but provides limited empirical evidence that these generated instances consistently preserve the original problem semantics and parameter relationships.

Questions for authors

Please see above

1 comment

The framework relies on LLM-inferred parameter domains and reconstructed problem descriptions, but provides limited empirical evidence that these generated instances consistently preserve the original problem semantics and parameter relationships.

Thank you for raising this concern about preserving problem semantics and parameter relationships. In Step (1), the mapping from problem parameters to LP coefficients is established through dynamic data-flow tracking during the execution of the formulation code. The LLM infers the semantic types and admissible ranges of parameters from the problem description, such as positive numbers, nonnegative numbers, positive integers, and proportions within [0,1][0,1]. In Step (3), we use the generated coefficients and the established mappings to construct a dependency graph over the affected problem parameters. We then prompt the LLM to update the affected numerical values in topological order, accounting for parameter dependencies while preserving the problem context and optimization task.

Using LLMs to extract parameter value information from natural language has precedents in prior work. For example, GPTuner [1] uses LLMs to transform natural-language tuning knowledge from sources such as database documentation into structured information, including recommended values and suggested lower and upper bounds. This provides a related precedent for using LLMs to assist in obtaining parameter ranges from textual descriptions.

To check output quality in our study, three graduate students manually reviewed the parameter ranges inferred by the LLM. No errors were found in the reviewed outputs. We also used GPT-5-mini to check the reconstructed problem descriptions, and this automated check detected no errors. We will clarify these checking procedures in Appendix C.8.

Reference

[1] Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. “GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization.” Proceedings of the VLDB Endowment, 17(8):1939–1952, 2024. doi:10.14778/3659437.3659449.

Reviewer_N6VAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a constraint-coverage-guided validation paradigm for automated linear programming modeling. Rather than evaluating automated models by checking whether the objective is correct, the authors focus on how many constraints the test instances exercise, since constraints that remain inactive may be omitted or misformulated without being detected. They propose a three-stage augmentation framework: (1) mapping problem parameters to formulation coefficients; (2) generating new LP instances via CGOS, Outer Frontier Search, and RHS/LHS Perturbation; and (3) reconstructing the coefficients back into the parameters of the original problem statement. They evaluate on five public datasets and show that the method is effective both at improving constraint coverage and at exposing hidden constraint-level modeling errors.

Critical strengths

The paper offers a different perspective on evaluating automated modeling: it focuses not only on objective correctness but also on constraint coverage. The proposed augmentation approach is feasible and can help uncover potential constraint-level modeling errors.

Critical weaknesses

The reference LPs used in the experiments strongly affect the results. If a reference LP contains redundant constraints that no instance can ever cover, then using that reference LP as the coverage target is unreasonable. Moreover, in real applications where no reference LP is available, the constraint coverage rate cannot be computed at all. As a direction for future work, I suggest the authors consider that constraints differ in importance, rather than treating coverage alone as the criterion.

Questions for authors
  1. Add some per-problem distribution analysis, Are the number of augmented instances of every problem same?
  2. Typo: DeepSeek V4(from Table 1)  DeepSeek V4 pro
4 comments

We sincerely thank you for reviewing our paper and providing us with valuable feedback. We have addressed your concerns as follows.

The reference LPs used in the experiments strongly affect the results. If a reference LP contains redundant constraints that no instance can ever cover, then using that reference LP as the coverage target is unreasonable.

Thank you for pointing out the effect of redundant constraints in the reference LP on the coverage criterion. We have revised and clarified it in Section Evaluation metrics. We exclude inequalities that are redundant in the original reference LP and whose LHS coefficients and RHS cannot be modified under the problem semantics. Four constraints meet this condition: one in NLP4LP, one in IndustryOR, and two in ReSocratic, with none in LogiOR or MAMO. Excluding these four constraints slightly reduces the denominator used to compute constraint coverage. We have accordingly corrected Figure 3, using the same adjusted criterion for the original and augmented test suites. We appreciate your careful observation.

The corrected Figure 3 results are summarized below. Instance counts for the augmented suite include the original instances.

DatasetOriginal countAugmented countOriginal coverage (%)Augmented coverage (%)
NLP4LP48926295
IndustryOR17317295
LogiOR21665295
MAMO707344598
ReSocratic184438365997

Moreover, in real applications where no reference LP is available, the constraint coverage rate cannot be computed at all. As a direction for future work, I suggest the authors consider that constraints differ in importance, rather than treating coverage alone as the criterion.

We agree that evaluating optimization models without a reference LP is a challenging and important problem, and establishing a reliable basis for evaluation in this setting is a valuable direction for future work. One related direction is to verify and refine optimization models during the modeling process without requiring a trusted reference LP. For example, Opt-Verifier [1] uses LLMs during modeling to back-translate an optimization model into an abstract modeling structure, compare it with the structure extracted from the problem description, and refine the model based on feedback. This line of work has a different research focus from ours, as we clarify below.

We recognize that the motivation for our study was not sufficiently clear in the original manuscript. Our goal is to evaluate the correctness of automatically generated optimization models, using reference models as ground truth. However, matching the reference model’s optimal objective value on a single instance does not establish modeling correctness, because omitted or incorrectly formulated constraints may leave that value unchanged. We address this limitation by augmenting the test suite with instances that exercise constraints the original instance leaves untested, thereby exposing modeling errors that agreement on the original answer conceals. The reference LP is the ground truth for this evaluation. It specifies the intended target constraints, which lets us check whether the test suite exercises them. We will clarify this research objective and the role of constraint coverage in evaluating modeling correctness in the Introduction (lines 49–52).

We also thank you for suggesting that constraints should be weighted according to their importance. Incorporating domain knowledge and risk-based weights could help prioritize testing and account for differences in constraint importance, which would improve the practical value of the evaluation.

Reference

[1] Haoyang Liu, Jie Wang, Boxuan Niu, Xiongwei Han, Yian Xu, Mingxuan Ye, Zijie Geng, Fangzhou Zhu, Tao Zhong, Mingxuan Yuan, and Jianye Hao. “Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification.” Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, 2026. arXiv:2605.29556.

Thank you for your careful reading of our paper. We address Q1 and Q2 below.

Q1. Add some per-problem distribution analysis, Are the number of augmented instances of every problem same?

The number varies across problems, as summarized below. The counts include only newly generated instances and exclude the original instances. Mean denotes the average number of newly generated instances per original problem, and Min–Max denotes the minimum and maximum numbers across original problems.

DatasetMeanMin–Max
NLP4LP0.920–3
IndustryOR0.820–3
LogiOR2.140–6
MAMO9.490–25
ReSocratic1.080–7

Our method does not generate a fixed number of instances for every problem. We have added the distributions of the number of generated instances per problem to Figure 5 in Appendix C.5. The number of instances in the generated test suite varies across original problems because they differ in both the number of constraints and their initial constraint coverage.

Q2. Typo: DeepSeek V4(from Table 1)

We have corrected “DeepSeek V4” to “DeepSeek V4 Pro” in Table 1.

Overall, the authors have addressed the concern about redundant constraints and provided a clearer explanation of the research scope. Nevertheless, the lack of differentiated constraint importance remain limitations of the proposed approach. These issues should be taken into account when assessing the generalizability and practical applicability of the method.

Thank you for acknowledging our correction concerning redundant constraints and our clarification of the research scope. We would like to further clarify that our central goal is to use constraint coverage to guide test instance generation, revealing constraint omissions or misformulations that remain undetected by answer comparison on the original instances. Each target constraint required by the problem should be included in testing, provided that it can be covered under admissible parameter settings, so that its correct formulation can be assessed. Accordingly, counting target constraints equally is consistent with our goal of evaluating modeling correctness.

Comments on the paper

2 comments

We thank the reviewers for their helpful feedback. The revised manuscript includes the following updates.

Main text

  • Introduction: Clarified the research motivation more clearly.
  • Related Work, Section 2.2: Added discussions of Opt-Verifier and Constraint Injection and clarified their differences from our approach.
  • Evaluation metrics and Figure 3: Clarified the definition of constraints coverage and corrected Figure 3. We thank reviewer N6VA for identifying this issue.

Appendix

  • C.2, Problem Size Distributions: Reports model size distributions across the five original datasets.
  • C.4, Additional Ablation Results: Reports results across five random seeds, examining test-suite size at matched coverage and the number of problems reaching the target coverage within the budget.
  • C.5, Distribution of Generated Instances: Reports the distribution of additional instance counts per original problem.
  • C.6, Generation Process Statistics: Reports CGOS solve counts, parameter perturbation counts, generation times, LLM costs, and new-witness rates.
  • C.7, Additional Evaluation on Bench4Opt: Reports model sizes, coverage improvements, and generation times on 66 additional continuous LP problems.
  • C.8, Output Quality Checks: Describes manual checks of inferred parameter domains and automated checks of reconstructed problem descriptions.

We sincerely thank the reviewers for their careful reading of our paper. We have responded to each review and summarize the main clarifications and additions below.

Clarification of evaluation metrics (Reviewer N6VA, Reviewer Exub). We further clarified the evaluation metric. We thank Reviewer N6VA for identifying the issue with redundant constraints. We refined the metric and corrected Figure 3 accordingly. In response to Reviewer Exub's concern, we clarified that coverage records whether a target constraint belongs to a linearly independent working set at an optimal solution, without requiring it to be indispensable to the current optimal value.

Experimental cost analysis (Reviewer N6VA, Reviewer Ctby). We added the detailed information on computational costs requested by the reviewer. Our method can substantially improve constraint coverage with modest computation, generating test instances that more thoroughly assess whether constraints are correctly modeled.

Clarification of the methodological framework (Reviewer Exub, Reviewer hihg). LLMs infer the semantic types and admissible ranges of parameters and reconstruct new problem descriptions. These tasks involve natural-language information extraction and text reconstruction, making them well suited to LLM assistance. Three graduate students manually checked the inferred parameter ranges, and GPT-5-mini checked the reconstructed descriptions, with no errors detected in the checked outputs.

Expanded experimental scale (Reviewer hihg). We conducted supplementary experiments on Bench4Opt, an LP modeling dataset containing larger models. Our method increased coverage from 30.30% to 95.75% with modest generation time. These results support extending our constraint coverage testing approach to larger LP models in existing modeling datasets.

Clarification of research motivation (Reviewer N6VA, Reviewer Exub, Reviewer Ctby, Reviewer hihg). We further clarified our research objective in the Introduction and explained how our approach differs from the related work raised by the reviewers. Our goal is to use reference models as ground truth and generate multiple numerical instances that cover different target constraints, providing a more thorough evaluation of generated formulations than answer comparison on a single instance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.