- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates errors in LLM-based operations research (OR) modeling, termed "optimization modeling hallucinations," where generated mathematical formulations (M) and solver code (S) fail to faithfully reflect the original natural-language problem description (P). The authors propose OPT-Kraken, a training-free, post-hoc verification and repair pipeline. The workflow first prompts an LLM to extract structured requirements and instance data from P, and runs directional consistency checks across P, M, and S to localize discrepancies. Based on the diagnosed error scope, it routes the problem to either solver-only repair, local model-and-code patching, or structural reconstruction. For local edits, the pipeline compiles a prompt contract from a handcrafted catalog of 24 OR-specific error types (RepairCards) to constrain the LLM's modifications and prevent regressions on already-correct components, followed by an execution and consistency audit. Evaluations across six OR word-problem benchmarks and multiple upstream generators show substantial accuracy gains with high rescue rates and low regression rates.
Critical strengths
- The paper targets an important failure mode in structured code generation: unlike syntax or runtime errors that can be caught by a compiler, semantic modeling errors in operations research still allow the solver (e.g., Gurobi) to execute cleanly and return an optimal solution to the wrong mathematical problem.
- Explicitly decoupling rescue rate (fixing wrong solutions) from regression rate (breaking already-correct solutions) is a sound evaluation principle for iterative self-refinement systems, and the contract-constrained editing mechanism effectively suppresses regressions compared to unconstrained re-prompting.
- The empirical evaluation is thorough within its target benchmark suite, covering both zero-shot pipelines and fine-tuned 7B/8B OR models across six datasets, supported by clear ablation studies on each stage of the pipeline.
Critical weaknesses
The local repair mechanism relies on a manually constructed catalog of 24 OR-specific error categories (RepairCards), such as flipped inequality directions, unit conversion mistakes, and spurious integrality constraints. Adapting this framework to other structured generation or program repair tasks would require domain experts to manually re-engineer a new taxonomy and corresponding contract rules.
The extraction of ground-truth proxies (requirements O and instance data D) and the directional fidelity checks are performed by prompting an LLM. The paper does not quantify the error rate of this extraction stage itself or analyze how errors in O and D propagate to downstream repairs. Furthermore, using a strong frontier model (DeepSeek V4 Flash) as the repair backbone for weaker upstream generators (such as 7B/8B fine-tuned models) confounds the architectural benefits of OPT-Kraken with the raw capability gap between the generator and the verifier.
The multi-stage workflow requires multiple sequential LLM calls for extraction, directional checking, contract compilation, repair, and auditing. The experimental comparison omits compute-matched test-time scaling baselines, such as best-of-N sampling or multi-sample consistency filtering under an equal token budget.
The evaluated benchmarks consist of short, self-contained word problems with small embedded parameters, leaving open how the dual-evidence extraction and directional checks scale to realistic industrial optimization workflows where problem logic and large-scale data tables are decoupled.
Questions for authors
What is the empirical accuracy of the initial O/D extraction stage, and how does OPT-Kraken perform when the repair backbone is restricted to the exact same model as the upstream generator (e.g., repairing Qwen2.5-7B outputs using Qwen2.5-7B)?
How does OPT-Kraken compare against a compute-matched best-of-N sampling baseline that uses the same total token budget per problem?
How were the 24 RepairCard categories constructed, and were any of these rules derived by inspecting failure cases on the test benchmarks?
7 comments
Evidence on O/D Extraction, Same Model Repair, Sampling Baselines, and RepairCards-2
2. Response to same model repair with a weaker backbone
Reviewer comment (Weakness): “Furthermore, using a strong frontier model (DeepSeek V4 Flash) as the repair backbone for weaker upstream generators (such as 7B/8B fine-tuned models) confounds the architectural benefits of OPT-Kraken with the raw capability gap between the generator and the verifier.”
Reviewer comment (Question): “How does OPT-Kraken perform when the repair backbone is restricted to the exact same model as the upstream generator (e.g., repairing Qwen2.5-7B outputs using Qwen2.5-7B)?”
A2: We thank the reviewer for raising this concern, which helps us assess OPT-Kraken under more limited model capacity. Table 4(a) in the main paper reports same-model generation and repair results for DeepSeek V4 Flash, Qwen3.8-27B, and GPT-5.6 Sol, with each model repairing its own zero-shot outputs. These three models are relatively capable, so the weaker-backbone setting warrants a separate evaluation.
To address this concern, we selected Qwen3-8B as a representative small backbone for an additional experiment on all 801 problems. Qwen3-8B generates the initial mathematical model and code in a zero-shot setting, then repairs them within OPT-Kraken using the same checkpoint. The overall results and results for each benchmark are now included in Appendix C.11 (Tables 18 and 19, pp. 33–34 of the revised manuscript).
Table 2: Same-model generation and repair on 801 problems. Generation and repair use the identical Qwen3-8B checkpoint.
| Setting | Accuracy | Rescue | Regression | Error Coverage |
|---|---|---|---|---|
| Qwen3-8B zero-shot | 18.0% (144/801) | — | — | — |
| Qwen3-8B + OPT-Kraken | 26.0% (208/801) | 10.2% (67/657) | 2.1% (3/144) | 78.5% (516/657) |
OPT-Kraken increases accuracy by 8.0 percentage points, correcting 67 initially incorrect artifacts while preserving 141 of the 144 initially correct results. Rescue is lower than in the stronger-backbone settings; we discuss the effect of backbone capability in the main-text limitations (Section 5.6, p. 10 of the revised manuscript). These results show that OPT-Kraken can repair a weaker 8B model's own outputs while preserving most initially correct results.
3. Response to best-of-N sampling under comparable token budgets
Reviewer comment (Question): “How does OPT-Kraken compare against a compute-matched best-of-N sampling baseline that uses the same total token budget per problem?”
A3: We thank the reviewer for this suggestion, which helps us assess how the compute budget affects repair performance. Appendix C.6, Table 13(a) reports the average total token usage of both upstream modeling frameworks and repair baselines: 34,614 for OptiMUS-0.3, 53,127 for SAC-Opt, and 12,774 for Multi-stage Prompt-based Repair.
We understand the reviewer's concern about matching compute budgets. We therefore evaluated a best-of-N multi-sample repair baseline on the same 801 zero-shot artifacts at a similar average token budget. The baseline generates multiple complete repair candidates from the same initial model and code. A generic task-based selector reads the problem and candidate artifacts, then chooses a candidate or retains the original. We use DeepSeek V4.1 Flash for this baseline because the original V4 Flash API has been retired. DeepSeek reports stronger performance for V4.1 Flash on several agent benchmarks. The baseline uses 33,646 tokens per problem on average, compared with 34,000 for OPT-Kraken. We have added its resource usage to Appendix C.6, Table 13 (p. 31 of the revised manuscript).
Table 3: Repair results on 801 DeepSeek zero-shot artifacts at similar average token cost.
| Method | Repair backbone | Initial accuracy | Final accuracy | Rescue | Regression | Mean total tokens |
|---|---|---|---|---|---|---|
| Multi-sample repair + task-based selection | DeepSeek V4.1 Flash | 44.8% (359/801) | 60.5% (485/801) | 29.4% (130/442) | 1.1% (4/359) | 33,646 |
| OPT-Kraken | DeepSeek V4 Flash | 44.8% (359/801) | 64.9% (520/801) | 36.9% (163/442) | 0.6% (2/359) | 34,000 |
With similar average token usage, OPT-Kraken achieves 7.5 percentage points higher Rescue and 4.4 points higher final accuracy, with fewer regressions. Its evidence-guided diagnosis and constrained repair correct more errors and preserve more initially correct outputs than this multi-sample baseline.
Evidence on O/D Extraction, Same Model Repair, Sampling Baselines, and RepairCards-3
4. Response to RepairCard construction and rule provenance
Reviewer comment (Question): “How were the 24 RepairCard categories constructed, and were any of these rules derived by inspecting failure cases on the test benchmarks?”
A4: We thank the reviewer for this question, which helps us clarify the construction of the RepairCard catalog and its relationship to the test benchmarks. We first drew on the optimization modeling process to identify potential errors that LLMs might make and constructed development data covering diverse optimization tasks. The development materials included verified correct problems and modeling artifacts, erroneous cases created through controlled modifications, and modeling artifacts naturally generated by multiple LLMs. We compared the problem descriptions, mathematical models, and solver code to analyze discrepancies in objectives, variables, constraints, data mapping, and code implementation.
From this analysis, we identified 83 fine-grained modeling hallucination types and documented their definitions, typical error forms, concrete examples, and possible consequences. We then grouped and refined related types according to their causes, category boundaries, and repair needs, yielding 24 categories. For each category, we developed a RepairCard specifying its applicability conditions, required evidence, permitted and prohibited edits, and acceptance criteria. No RepairCard rules were derived from inspecting failure cases on the test benchmarks.
5. Response to RepairCard scope and cross-domain adaptation
Reviewer comment (Weakness): “The local repair mechanism relies on a manually constructed catalog of 24 OR-specific error categories (RepairCards), such as flipped inequality directions, unit conversion mistakes, and spurious integrality constraints. Adapting this framework to other structured generation or program repair tasks would require domain experts to manually re-engineer a new taxonomy and corresponding contract rules.”
A5: We thank the reviewer for raising this question about the scope and portability of RepairCards. The 24 categories describe recurring errors in the optimization modeling process, rather than errors specific to a particular application domain. They cover errors in objectives, variables, constraints, data binding, model-to-code translation, and result extraction. The same frozen catalog is used across the six optimization benchmarks and multiple upstream generators in our evaluation, without constructing a separate taxonomy for each benchmark.
RepairCards guide local repair. When source-grounded evidence indicates that local edits are insufficient, OPT-Kraken can use structural reconstruction. Eligible remaining errors can also receive a direct repair followed by an independent audit. These paths allow repair attempts for errors that may not fit a local RepairCard; success still depends on the available evidence and the model's capabilities.
Our study focuses on hallucination repair for LLM-based optimization modeling, consistent with the Optimization area of our submission. RepairCards use optimization modeling knowledge to define edit boundaries and acceptance criteria. Extending the evidence-guided repair and auditing mechanisms to other structured generation or program repair tasks is a direction for future research. In response to your suggestion, Section 6 (p. 10 of the revised manuscript) now outlines future work on using domain knowledge from areas such as power system optimization to strengthen evidence extraction and repair verification.
Evidence on O/D Extraction, Same Model Repair, Sampling Baselines, and RepairCards-1
Dear Reviewer 334a:
We thank the reviewer for the thoughtful review and for recognizing the importance of semantic modeling errors and the value of evaluating rescue and regression separately. Your comments have helped us sharpen the evaluation of OPT-Kraken, particularly the reliability of extracted evidence and the benefits of structured repair relative to additional sampling. Below, we provide expert-validated O/D extraction results, a same-model 8B repair experiment, and an additional best-of-N repair comparison to address these concerns directly.
Please see below for our point-by-point responses to your comments and questions. Changes in the revised manuscript are highlighted in blue.
1. Response to empirical accuracy and downstream effects of initial O/D extraction
Reviewer comment (Question): “What is the empirical accuracy of the initial O/D extraction stage?”
Reviewer comment (Weakness): “The paper does not quantify the error rate of this extraction stage itself or analyze how errors in O and D propagate to downstream repairs.”
A1: We thank the reviewer for requesting a direct empirical assessment of the initial O/D extraction stage and its effect on subsequent repair. This helps us assess how accurately the extracted requirements and data reflect the source problem, and what happens when they are used in repair. Three experts with PhDs in mathematics and expertise in operations research and optimization manually checked the archived initial extractions against the original problems. The review covered all 801 problems for each of DeepSeek V4 Flash and Qwen3.8-27B. We have added the extraction accuracy results to Appendix C.10 (Table 17, p. 33 of the revised manuscript).
We measure extraction accuracy separately for O and D as the fraction of problems whose extraction contains no substantive error relative to the original problem. For D, we evaluate the validated data manifest and count empty extractions as errors. Every rate uses all 801 problems as its denominator. The joint O/D pass rate requires both components to pass on the same problem.
Table 1: Problem level empirical accuracy of initial O/D extraction. All rates use the full 801-problem cohort per backbone; empty D extractions count as errors.
O and D show high source consistency, with D more accurate for each backbone. Both extractions pass in 87.3% of the DeepSeek cases and 81.3% of the Qwen cases, providing source-consistent evidence for OPT-Kraken's subsequent diagnosis and repair.
To illustrate how errors in O and D can propagate to downstream repairs, we have added a concrete trace for MAMO Complex-112 in Appendix C.10 (p. 33 of the revised manuscript). The extraction interprets “144 grams of protein” as a carbohydrate requirement, and the repair candidate uses that interpretation in a constraint. The LLM audit returns PASS. A separate evidence check, however, finds that the candidate also changes integer variables to continuous variables without sufficient source support, so the candidate is not committed. This example traces an extraction error into a repair candidate and illustrates OPT-Kraken's separation of candidate generation and evidence-based acceptance.