- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that matching the reference objective value is not a reliable correctness test for LLM-generated optimization models. It defines optimization-modeling hallucinations as semantic deviations of the formulation or solver code from the problem description, and proposes an 83-entry taxonomy spanning objective, variable, constraint and implementation errors. It builds development and test suites with over 12,000 instances, organized into three test sets: reference-clean artifacts for false alarms, controlled single-error mutations for localization, and unedited outputs from 13 generators for natural-error detection. It then evaluates six LLM-based detectors, including a multi-agent detector called OptArgus.
Critical strengths
The problem is real and well motivated. Among outputs that match the reference objective value, 5.6% to 27.1% still contain structural errors (Table 1). The three-set evaluation design, which separates false-alarm restraint, localization and natural-error detection, is a sensible structure for this task.
Critical weaknesses
- The benchmark labels are not validated. They come from model-assisted review with no inter-annotator agreement. Detectors are scored against outdated labels, and the scores were not recomputed on the revised ones (App. B.4.1).
- The detectors only modestly beat a trivial baseline. With a 33% error rate (2084/6292, Table 8), flagging everything gives a Halluc-F1 of about 0.50. Single-Agent reaches 0.52 and OptArgus 0.62 (Table 3).
- On two generators, OptMATH and SIRL, OptArgus reaches recall 1.0 at precision 0.289 and 0.281, which equal those generators’ error rates (Table 7). It therefore flags every output, correct or not, and its F1 there matches the flag-all baseline.
- OptArgus flags 40.6% of valid equivalent reformulations as errors, which contradicts the paper’s own definition of a hallucination (App. C.4).
- OptArgus’s lead over detectors with the same compute budget is very small, such as 0.403 against 0.401 on Type@1. It is never tested for significance (Table 3, App. B.3.3).
Questions for authors
- How much of the natural-set labeling was done by humans rather than models, and was any of it independently checked?
- Why were the detector scores not recomputed after the labels were revised?
- Which detector selected the reference-clean set, and is it related to OptArgus?
- How do you reconcile the 40.6% false-positive rate on valid reformulations with the paper’s semantic definition of a hallucination?