acceptodds
Under review as a conference paper at ICLR 2027

Hallucinations in LLM-Based Optimization Modeling: Definition, Taxonomy, Detection Benchmarks, and Reference Detectors

Abstract

Large language models (LLMs) are increasingly used to translate natural-language optimization problems into mathematical formulations and solver code. Most existing work evaluates the resulting models by comparing the objective value produced by generated solver code with a reference value. However, matching the reference objective value is not a reliable test of correctness, as it can mask unintended changes to optimization semantics, such as changes to the intended decisions, feasible region, objective meaning, or solver behavior. We define an optimization-modeling hallucination as a deviation from the intended optimization semantics. We develop, to our knowledge, the first fine-grained hallucination taxonomy specifically for optimization modeling, spanning objective, variable, constraint, and implementation failures. Building on this definition and taxonomy, we formulate optimization-modeling hallucination detection as structured prediction over tuples comprising a problem description, symbolic model, and solver code. To support detector development and evaluation, we construct development and test suites comprising more than 12,000 detection instances, with disjoint source-problem sets. The test suite contains correct examples for measuring false alarms, controlled changes for measuring localization, and LLM-generated outputs for measuring detection of naturally occurring errors. Using these suites, we design and evaluate six detectors for optimization-modeling hallucinations. Among them, OptArgus is a multi-agent detector in which separate agents inspect the objective, variables, constraints, and solver code before combining their findings into a final diagnosis. Experiments show that OptArgus offers a strong overall trade-off across the three evaluation settings, combining the highest controlled-error localization scores among the six detectors with competitive false-alarm control and natural-error detection, while producing concise diagnoses. Together, these contributions establish optimization-modeling hallucination detection as a concrete empirical problem and foundation for more reliable optimization modeling.

Primary area:
optimization
Keywords:
Optimization Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FbsuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that matching the reference objective value is not a reliable correctness test for LLM-generated optimization models. It defines optimization-modeling hallucinations as semantic deviations of the formulation or solver code from the problem description, and proposes an 83-entry taxonomy spanning objective, variable, constraint and implementation errors. It builds development and test suites with over 12,000 instances, organized into three test sets: reference-clean artifacts for false alarms, controlled single-error mutations for localization, and unedited outputs from 13 generators for natural-error detection. It then evaluates six LLM-based detectors, including a multi-agent detector called OptArgus.

Critical strengths

The problem is real and well motivated. Among outputs that match the reference objective value, 5.6% to 27.1% still contain structural errors (Table 1). The three-set evaluation design, which separates false-alarm restraint, localization and natural-error detection, is a sensible structure for this task.

Critical weaknesses
  1. The benchmark labels are not validated. They come from model-assisted review with no inter-annotator agreement. Detectors are scored against outdated labels, and the scores were not recomputed on the revised ones (App. B.4.1).
  2. The detectors only modestly beat a trivial baseline. With a 33% error rate (2084/6292, Table 8), flagging everything gives a Halluc-F1 of about 0.50. Single-Agent reaches 0.52 and OptArgus 0.62 (Table 3).
  3. On two generators, OptMATH and SIRL, OptArgus reaches recall 1.0 at precision 0.289 and 0.281, which equal those generators’ error rates (Table 7). It therefore flags every output, correct or not, and its F1 there matches the flag-all baseline.
  4. OptArgus flags 40.6% of valid equivalent reformulations as errors, which contradicts the paper’s own definition of a hallucination (App. C.4).
  5. OptArgus’s lead over detectors with the same compute budget is very small, such as 0.403 against 0.401 on Type@1. It is never tested for significance (Table 3, App. B.3.3).
Questions for authors
  1. How much of the natural-set labeling was done by humans rather than models, and was any of it independently checked?
  2. Why were the detector scores not recomputed after the labels were revised?
  3. Which detector selected the reference-clean set, and is it related to OptArgus?
  4. How do you reconcile the 40.6% false-positive rate on valid reformulations with the paper’s semantic definition of a hallucination?
Reviewer_MTnkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper defines optimization-modeling hallucinations as deviations from the intended semantics across a problem description, symbolic model, and solver implementation. It introduces an 83-entry taxonomy covering objective, variable, constraint, and implementation issues, while distinguishing semantic errors from equivalent reformulations and quality warnings. The authors construct development and test suites containing more than 12,000 artifacts and evaluate six detector designs. OptArgus uses a conductor and specialized agents to inspect model components and consolidate their findings. It obtains the strongest controlled-case taxonomy scores among the six main detectors, while a generic taxonomy-aware verifier performs best on the natural-output aggregate metrics. The paper also reports ablations, inference costs, an equivalence stress test, and an exploratory repair study.

Critical strengths

S1. The paper provides a useful semantic framing and detailed vocabulary for optimization-model auditing.

The distinction among consistency of the problem, symbolic model, and implementation; objective-value agreement; equivalent reformulations; and quality warnings is important. The 83-entry hierarchy is more diagnostic than a binary correctness label and could support future annotation and detector development. (Sections 3-5; Appendix D)

S2. The resource is broad, auditable, and unusually transparent about its limitations.

The three test settings cover restraint on clean artifacts, diagnosis of controlled changes, and naturally occurring errors. The release includes data, detector configurations and code, ablations, cost measurements, and an equivalence stress test. The paper also openly reports selection bias, incomplete catalog coverage, residual errors, label-version differences, and annotation limitations. (Section 7; Appendices B and C)

Critical weaknesses

W1. The labels and evaluated artifacts are not sufficiently stable or consistently versioned to support the main detector comparisons.

For the reference-clean set, cases were selected using an earlier detector, five references were later revised without updating the released clean files, and the archived reports lack input snapshots or hashes. The released clean case resocratic_00085 also appears incorrect: the problem permits up to 40 units of product 1 from 120 units of material at 3 units per product, but the model and code impose an upper and activation bound of 20 while labeling the case clean. If this tuple was evaluated, a valid finding would be counted against detector restraint. (Appendix B.2.1, pp. 19-20)

The controlled set realizes only 58 of 83 catalog entries, includes 33 quality-warning cases in the main scores, and does not guarantee that each artifact contains only its designated error. There also appear to be target-label mismatches. For example, industryor_00000__constraint_drop_cardinality_upper deletes the maximum-cardinality constraint but is labeled Wrong Aggregation Level, whose taxonomy definition concerns pooled versus per-object enforcement. A scan of the released manifest finds 30 similarly labeled constraint-deletion cases. (Appendix B.3, pp. 20-21; Appendix D, entry 3.6.1)

For the natural set, Tables 1 and 6 use revised labels while Table 3 retains historical labels. The paper reports that reviewing 94 previously null GLM records added 41 positives, changing the count from 107 to 148, but detector scores were not recomputed. No independent double annotation or inter-rater agreement is reported, and the released artifact does not appear to include the revised labels for the 7,744-output audit. These issues affect all three principal evaluation settings and require a frozen, released, independently audited label set followed by recomputation of the main results. (Appendix B.4.1, pp. 22-23)

W2. The metrics do not evaluate the complete task defined by the paper, and the semantic-equivalence results weaken the central detector claim.

Section 5 defines prediction over affected element-type pairs with supporting evidence. However, the controlled metrics compare only the first finding's taxonomy labels, without scoring the target element or validating its explanation. Natural-set metrics operate only at artifact and major-category levels. The reported results therefore measure label agreement rather than element-level localization. OptArgus's Type@1 is 0.403, nearly tied with the generic taxonomy-aware verifier at 0.401. Moreover, the generic verifier has higher natural-set Halluc-F1, Macro-F1, and Micro-F1 than OptArgus. The claimed overall trade-off therefore needs a predefined criterion. (Section 5; Table 3; Appendix B.1)

The equivalence stress test is especially important because valid reformulations are excluded from the paper's hallucination definition. OptArgus detects 85.9% of non-equivalent rewrites but also flags 40.6% of equivalent rewrites, and handles both members correctly in only 46.9% of pairs. This shows that the current detector often fails to distinguish a valid rewrite from its minimally invalid counterpart. The results support using OptArgus as an auditing aid, but not a strong claim of reliable semantic discrimination. (Appendix C.4, Table 16)

Questions for authors

Q1. Can the authors release one frozen label version and recompute all main results? This should include hashes or snapshots of evaluated inputs, the revised clean cases, and the revised natural-output annotations.

Q2. Can the controlled set be re-audited for target-label correctness and error isolation? In particular, why is deletion of cardinality_upper labeled Wrong Aggregation Level, and how were the other similarly constructed cases validated?

Q3. Can the authors evaluate exact affected-element localization and evidence correctness? Taxonomy-label accuracy alone does not evaluate the structured task defined in Section 5.

Q4. What detector claims remain justified by the equivalence results? Can the authors narrow the claims and compare with dedicated semantic-equivalence or falsification approaches?

Reviewer_Y9CMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper defines optimization-modeling hallucinations as semantic inconsistencies across a problem description, a symbolic formulation, and solver code. It proposes a fine-grained taxonomy and development/test suites covering reference-clean artifacts, controlled mutations, and natural model outputs. Six reference detectors, including the specialist-based OptArgus system, illustrate tradeoffs among reporting restraint, diagnosis, and natural-error detection.

Critical strengths
  1. The paper makes the important distinction between matching an objective value and preserving the intended optimization semantics. The taxonomy's treatment of equivalent reformulations and nonsemantic quality warnings offers a useful basis for more meaningful evaluation.

  2. The separate clean, controlled, and natural settings, together with several approximately budget-matched detectors, expose different failure modes. Reporting unfavorable equivalence-test results and stronger natural-set baselines adds value beyond promoting a single architecture.

Critical weaknesses
  1. Known label defects and unresolved input versions undermine the benchmark comparisons. Appendix B.4.1 states that 94 previously null GLM-4.5-Air records were treated as negatives in the historical evaluation. A later audit found 41 positives among them, but detector scores have not been recomputed. Correct findings can therefore be penalized as false positives, potentially affecting methods differently. Appendix B.2.1 also reports five revised clean reference artifacts whose corrections are absent from the released files, while archived OptArgus runs lack input snapshots or hashes establishing exact correspondence. These are documented defects, not hypothetical annotation noise. A consistent reviewed label snapshot and fixed reference inputs are needed before the results can establish detection performance and false-alarm control.

  2. The controlled metric does not establish the claimed localization quality. Equations (11)–(13) score the first finding's taxonomy label without verifying its location or explanation. Appendix B.3.1 also documents cases with additional unannotated errors: in the ReSocratic seed00085 example, a designated activation-bound change coexists with an incorrect raw-material coefficient. A detector reporting that other genuine error first can lose credit, while a finding with the expected label but an incorrect location can receive credit. The scores therefore measure agreement with a designated mutation label, rather than recovery of the element–type pairs in the formal task. A semantically audited subset with isolated errors, or annotations that admit all valid findings, would connect the evaluation more closely to the intended task.

Questions for authors
  1. Can the existing detector outputs be rescored against a consistent, reviewed label snapshot, with unknown labels handled separately and all clean-set inputs tied to the corrected reference version?

  2. On audited controlled cases, what conclusions remain when the affected element and supporting evidence must be correct as well as the taxonomy label?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.