acceptodds
Under review as a conference paper at ICLR 2027

Changes in Problem Presentation Can Break LLM-Based Optimization Modeling

Abstract

Optimization modeling is the process of translating natural language descriptions of optimization problems into mathematical formulations and solver code. Large language models (LLMs) are increasingly used to automate this process. In practice, users may describe the same problem in different ways. LLM-based modeling systems must handle these different but equivalent problem descriptions correctly, especially in high-stakes applications such as healthcare and energy. Yet we empirically found that LLMs can generate correct mathematical formulations and solver code for an optimization problem, but fail when the same problem is described differently. This raises a central question: how robust is LLM-based optimization modeling across different natural-language descriptions of the same problem? However, existing optimization modeling benchmarks focus primarily on accuracy but provide limited evidence of robustness to such changes in natural-language descriptions. To address this gap, we introduce RobustOpt, an extensible framework with two complementary benchmarks for requirement understanding through question answering (QA) and end-to-end optimization modeling (Coding) to quantify this robustness. Specifically, we systematically construct and validate alternative problem descriptions that preserve the underlying optimization problem, then compare LLM performance across these descriptions. On each benchmark, we evaluate the robustness of 14 LLM-based systems: seven based on general-purpose LLMs and seven on models specifically fine-tuned for optimization modeling. The results show that even fine-tuned models can fail on 34.19% of new descriptions of problems they initially handle correctly; failures reach 67.40% among general-purpose models. Further error analysis reveals mistakes in specifying decision variables, objectives, and constraints. These findings highlight robustness to problem presentation as a key requirement for trustworthy LLM-based optimization modeling, motivating its explicit integration into model evaluation and training.

Primary area:
optimization
Keywords:
Optimization Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NKjROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers the following robustness question: Do LLM-based optimization modelling systems remain correct when the same underlying optimization problem is presented in different ways. In particular, the authors introduce RobustOpt, a framework for constructing alternative problem descriptions that preserve that are essentially equivalent to the source problem - i.e., should result in the same decision variables, constraints and objectives. The framework contains two benchmarks: RobustOpt-QA which evaluates the understanding of optimization requirements through multiple-choice questions, and RobustOpt-Coding which evaluates the modeling itself, namely the solver code generated by the LLM. Using the framework, the paper evaluates multiple genera-purpose LLMs as well as fine-tuned LLMs for optimization. The authors find that the models fail on alternative descriptions of problems they previously handled correctly. They also analyze the resulting modeling errors and propose some defenses for increasing accuracy.

Critical strengths

Thank you for submitting your work to ICLR. Here are the paper's main strengths:

  1. Identifying robustness issues with respect to problem description. The paper shows that LLM-based optimization systems can lose correctness when the presentation of a problem changes, even when the underlying optimization problem remains the same. This exposes an important reliability and robustness gap with implication on practical accuracy of these systems.
  2. Non-trivial transformations in RobustOpt-Coding. RobustOpt-Coding constructs alternative versions of a problem by applying diverse transformations to its description and associated data while preserving the intended optimization model. Examples include distributing related information across tables or reports, converting linked units, and requiring coefficients to be recovered from measurements. This makes the benchmark rich and challenging as well as interesting from a practical perspective.
Critical weaknesses
  1. Incomplete verification of semantic equivalence and formulation correctness. The paper defines equivalent descriptions as preserving the complete feasible set and objective function, but as far as I can tell its validation procedures do not fully establish this property, particularly for complex transformations. Similarly, the primary Coding metric checks execution and agreement with a reference numerical result, which does not guarantee that the generated formulation is correct. The formulation-level analysis covers only selected outputs and is explicitly not a complete audit. To be fair, I think it's a very complicated problem on its own, so I would have appreciated a more careful exposition of the methods to address it, as well as an explicit accounting of existing current limitations.
  2. Limited machine-learning contribution. The paper is primarily a benchmark and evaluation study. It would have been great to see some more effort on the ML side, e.g., fine-tuning method for improving robustness based on the study. The proposed defenses are preliminary inference-time interventions: structured self-checking for QA and selecting a second candidate after execution failure for Coding. These experiments are somewhat limited in scope and do not feel rigorous or methodological enough for confronting robustness.

On top of these concrete weaknesses, I believe that the presentation of the paper can be greatly improved. It was hard to follow some of the technical parts (e.g., metric definitions). The conclusion and limitation section does not add much and has to be proof-read.

Questions for authors
  1. Section 4.1, description of benchmark construction. "...into 1,200 original and perturbed question-context presentation": It is not clear to me whether you perturbed only the description of the problem or both the description and the way the multiple-choice question is asked. Please clarify.
  2. Metrics (Sec. 4.2). I found the description of metrics somewhat hard to follow, although I believe that I eventually understood the details. That said, there are 4 metrics for QA and 6 for coding. Can the authors please comment which are the key ones, and whether they can have a unified set of metrics for both benchmarks?
  3. Verification of input equivalence. Following on my comment from above, how do the authors verify that the more complex RobustOpt-Coding transformations preserve the complete feasible set and objective function, rather than only selected fields or the optimal value? The authors do state that "“The numerical checks and checks of selected formulations reported in this paper do not establish this full criterion” (Appendix E.2). Nonetheless, please elaborate more. For example, what proportion of these inputs receives a complete mathematical equivalence check.
Reviewer_tno9OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the semantic robustness of LLM-based optimization modeling, focusing on whether LLMs can consistently formulate and solve the same optimization problem when its natural-language description changes. The authors propose RobustOpt, an extensible evaluation framework comprising two complementary benchmarks: RobustOpt-QA for assessing the understanding of optimization requirements and RobustOpt-Coding for evaluating end-to-end mathematical modeling and solver-code generation. The framework constructs alternative problem descriptions through controlled transformations while attempting to preserve the underlying optimization problem. It further introduces paired evaluation metrics to measure correctness retention and identify failures that may be overlooked by conventional average accuracy. Experiments involving 14 general-purpose and optimization-specialized LLMs reveal substantial performance variations across equivalent problem descriptions. Additional analyses examine the effects of different transformations, errors in mathematical formulations, and potential mitigation strategies. The results highlight the importance of semantic robustness in evaluating the reliability of LLM-based optimization modeling systems.

Critical strengths
  1. The paper addresses an important but insufficiently examined issue in LLM-based optimization modeling: whether a model can consistently formulate the same optimization problem under semantically equivalent descriptions. This is particularly relevant to practical applications, where users may express identical requirements in different ways. The distinction between average accuracy and correctness retention provides a meaningful perspective on evaluating the reliability of LLM-based optimization systems.

  2. The proposed RobustOpt framework offers a relatively comprehensive evaluation methodology by combining requirement-understanding assessment and end-to-end optimization modeling. The experiments cover 14 LLM-based systems, multiple optimization problem classes, and different types of description transformations. The paired evaluation design and additional analysis of mathematical formulation errors strengthen the empirical contribution and provide useful insights for future research.

Critical weaknesses
  1. The reliability of the benchmark depends critically on whether the transformed descriptions preserve the original optimization problems. Although the paper provides a systematic construction and validation procedure, semantic equivalence is difficult to establish rigorously, particularly for nonlinear problems, unit conversions, and descriptions involving scattered or implicit information. The current validation approach does not fully eliminate the possibility that some observed failures arise from ambiguity or unintended changes in problem requirements rather than limitations of the evaluated LLMs.

  2. The evaluation methodology does not always establish whether the generated mathematical formulations are correct. The primary Coding evaluation relies on successful execution and agreement with reference numerical results, which are insufficient to guarantee mathematical equivalence. Although the authors acknowledge this limitation and introduce supplementary formulation checks, these checks are applied selectively. Consequently, the reported robustness results should be interpreted primarily as numerical-output robustness rather than a complete assessment of mathematical modeling robustness.

Questions for authors
  1. How is semantic equivalence guaranteed for the transformed optimization problems, particularly when multiple transformations are combined or when the original problems involve nonlinear constraints? Could the authors provide more quantitative evidence of the reliability of the validation procedure, such as independent human agreement or a detailed analysis of rejected and ambiguous transformations?

  2. The proposed framework defines equivalent optimization problems in terms of identical feasible sets and objective functions. However, the primary Coding evaluation relies on numerical agreement with reference solutions. To what extent do the authors believe this evaluation captures the proposed notion of semantic robustness? Could additional formulation-level verification be conducted on a larger and more representative subset?

  3. The paper builds on existing benchmarks and established robustness-testing principles. Could the authors further clarify the methodological novelty of RobustOpt compared with existing work on equivalent problem representations, robustness evaluation, and optimization modeling benchmarks? In particular, what capabilities does the proposed framework offer that cannot be evaluated using existing perturbation-based benchmarking approaches?

  4. The experimental results indicate that some LLMs exhibit substantial failure rates under equivalent descriptions. However, how much of this variation can be attributed specifically to the transformations rather than inherent generation variability? Although repeated-input experiments are provided, a more systematic comparison across models and transformation categories would strengthen the conclusions.

  5. How representative are the selected 200 Coding problems and 200 QA questions of practical optimization modeling tasks? In particular, could the authors clarify whether the observed robustness patterns remain consistent across different problem scales, mathematical structures, and levels of modeling complexity?

  6. The paper evaluates two relatively simple mitigation strategies, including requirement checking and generating an additional candidate. Could the authors provide a more systematic analysis of whether these strategies improve correctness retention, rather than only average accuracy or execution success, and discuss their additional computational costs?

Reviewer_jAcmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focus on whether the LLM modeling is robust if the same problem described in different ways. It formalizes this topic, gives two benchmarks, and then find the LLMs had massive failures after a series of tests of 14 LLMs. Some other findings are shown like errors occur in both variables, objectives, and constraints, and average accuracy can stay almost unchanged as some answers turn wrong and others turn right.

Critical strengths

The paper correctly formalizes the problem and reveals a blind spot that mathematical modeling may be not so robust for current LLMs. The benchmarks and measurements can help to better develop LLM modeling.

Critical weaknesses
  • W1. Though the abstract sentences a big failure like “even fine-tuned models can fail on 34.19% of new descriptions of problems”, but I find that repeated responses can yield a comparable and sometimes even bigger inconsistent number, under B. Same options in both comparisons. So the effect of changes in problem is not isolated from sampling noise, and I also don’t see it isolated from the length-induced difficulty, so it looks not really significant especially for some LLMs.

  • W2. The conceptual contribution is incremental, the conclusions are not particularly novel to me, and there is no downstream validation like “so if we use xxx metric to select/train model, the actual performance is xx% better than using accuracy alone”. They constraints the contribution.

Questions for authors
  • Q1. Can you decompose the failure into repeat sampling noise + length-induced difficulty + problem change effect and report the net effect for each LLM?
  • Q2. The QA Panel B contain only 29/21 questions, can you add more cases from the 200 problems that make the coverage more evenly? and the coding part contains only one pair of repeat generations each problem, is it too coarse to be a no-change baseline?
  • Q3. Can you demonstrate a downstream validation that what we can do better with the measurement of robust modeling?

Comments on the paper

1 comment

Dear Reviewers,

Please note that the author rebuttal is now available. We kindly ask you to read it, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.

Best regards

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.