- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers the following robustness question: Do LLM-based optimization modelling systems remain correct when the same underlying optimization problem is presented in different ways. In particular, the authors introduce RobustOpt, a framework for constructing alternative problem descriptions that preserve that are essentially equivalent to the source problem - i.e., should result in the same decision variables, constraints and objectives. The framework contains two benchmarks: RobustOpt-QA which evaluates the understanding of optimization requirements through multiple-choice questions, and RobustOpt-Coding which evaluates the modeling itself, namely the solver code generated by the LLM. Using the framework, the paper evaluates multiple genera-purpose LLMs as well as fine-tuned LLMs for optimization. The authors find that the models fail on alternative descriptions of problems they previously handled correctly. They also analyze the resulting modeling errors and propose some defenses for increasing accuracy.
Critical strengths
Thank you for submitting your work to ICLR. Here are the paper's main strengths:
- Identifying robustness issues with respect to problem description. The paper shows that LLM-based optimization systems can lose correctness when the presentation of a problem changes, even when the underlying optimization problem remains the same. This exposes an important reliability and robustness gap with implication on practical accuracy of these systems.
- Non-trivial transformations in RobustOpt-Coding. RobustOpt-Coding constructs alternative versions of a problem by applying diverse transformations to its description and associated data while preserving the intended optimization model. Examples include distributing related information across tables or reports, converting linked units, and requiring coefficients to be recovered from measurements. This makes the benchmark rich and challenging as well as interesting from a practical perspective.
Critical weaknesses
- Incomplete verification of semantic equivalence and formulation correctness. The paper defines equivalent descriptions as preserving the complete feasible set and objective function, but as far as I can tell its validation procedures do not fully establish this property, particularly for complex transformations. Similarly, the primary Coding metric checks execution and agreement with a reference numerical result, which does not guarantee that the generated formulation is correct. The formulation-level analysis covers only selected outputs and is explicitly not a complete audit. To be fair, I think it's a very complicated problem on its own, so I would have appreciated a more careful exposition of the methods to address it, as well as an explicit accounting of existing current limitations.
- Limited machine-learning contribution. The paper is primarily a benchmark and evaluation study. It would have been great to see some more effort on the ML side, e.g., fine-tuning method for improving robustness based on the study. The proposed defenses are preliminary inference-time interventions: structured self-checking for QA and selecting a second candidate after execution failure for Coding. These experiments are somewhat limited in scope and do not feel rigorous or methodological enough for confronting robustness.
On top of these concrete weaknesses, I believe that the presentation of the paper can be greatly improved. It was hard to follow some of the technical parts (e.g., metric definitions). The conclusion and limitation section does not add much and has to be proof-read.
Questions for authors
- Section 4.1, description of benchmark construction. "...into 1,200 original and perturbed question-context presentation": It is not clear to me whether you perturbed only the description of the problem or both the description and the way the multiple-choice question is asked. Please clarify.
- Metrics (Sec. 4.2). I found the description of metrics somewhat hard to follow, although I believe that I eventually understood the details. That said, there are 4 metrics for QA and 6 for coding. Can the authors please comment which are the key ones, and whether they can have a unified set of metrics for both benchmarks?
- Verification of input equivalence. Following on my comment from above, how do the authors verify that the more complex RobustOpt-Coding transformations preserve the complete feasible set and objective function, rather than only selected fields or the optimal value? The authors do state that "“The numerical checks and checks of selected formulations reported in this paper do not establish this full criterion” (Appendix E.2). Nonetheless, please elaborate more. For example, what proportion of these inputs receives a complete mathematical equivalence check.
Dear Reviewers,
Please note that the author rebuttal is now available. We kindly ask you to read it, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.
Best regards
AC