acceptodds
Under review as a conference paper at ICLR 2027

Fix What's Wrong, Keep What's Right: Hallucination Repair for LLM-Based Optimization Modeling

Abstract

Large language models (LLMs) are increasingly being explored to automate operations research modeling, reduce modeling time and lower the expertise barrier to optimization based decision support. However, this process often introduces inconsistencies between generated artifacts and the original problem, which we refer to as optimization modeling hallucinations. Despite progress in general-purpose hallucination mitigation, optimization modeling poses a distinct repair challenge: restoring fidelity to the original problem through coordinated model–code updates. We introduce OPT-Kraken, a training-free hallucination repair framework. To the best of our knowledge, it is the first post-hoc framework for repairing hallucinations in LLM-based optimization modeling artifacts. It extracts problem requirements and instance data, localizes errors by checking problem–model fidelity, model–code consistency, and data grounding, and selects repair granularity by the affected scope. For local repair, category rules and instance evidence are compiled into contracts specifying permitted edits, expected effects, and protected content. Experiments on established optimization modeling benchmarks show positive gains across automatic modeling frameworks and fine-tuned models. On zero-shot artifacts from multiple base models, OPT-Kraken effectively repairs optimization modeling hallucinations while preserving initially correct artifacts. These results demonstrate effective and controlled repair for LLM-based optimization modeling artifacts.

Primary area:
optimization
Keywords:
Large Language Models, Optimization Modeling, Operations Research, Hallucination Repair, Evidence-Constrained Repair

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_334aOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates errors in LLM-based operations research (OR) modeling, termed "optimization modeling hallucinations," where generated mathematical formulations (M) and solver code (S) fail to faithfully reflect the original natural-language problem description (P). The authors propose OPT-Kraken, a training-free, post-hoc verification and repair pipeline. The workflow first prompts an LLM to extract structured requirements and instance data from P, and runs directional consistency checks across P, M, and S to localize discrepancies. Based on the diagnosed error scope, it routes the problem to either solver-only repair, local model-and-code patching, or structural reconstruction. For local edits, the pipeline compiles a prompt contract from a handcrafted catalog of 24 OR-specific error types (RepairCards) to constrain the LLM's modifications and prevent regressions on already-correct components, followed by an execution and consistency audit. Evaluations across six OR word-problem benchmarks and multiple upstream generators show substantial accuracy gains with high rescue rates and low regression rates.

Critical strengths
  1. The paper targets an important failure mode in structured code generation: unlike syntax or runtime errors that can be caught by a compiler, semantic modeling errors in operations research still allow the solver (e.g., Gurobi) to execute cleanly and return an optimal solution to the wrong mathematical problem.
  2. Explicitly decoupling rescue rate (fixing wrong solutions) from regression rate (breaking already-correct solutions) is a sound evaluation principle for iterative self-refinement systems, and the contract-constrained editing mechanism effectively suppresses regressions compared to unconstrained re-prompting.
  3. The empirical evaluation is thorough within its target benchmark suite, covering both zero-shot pipelines and fine-tuned 7B/8B OR models across six datasets, supported by clear ablation studies on each stage of the pipeline.
Critical weaknesses

The local repair mechanism relies on a manually constructed catalog of 24 OR-specific error categories (RepairCards), such as flipped inequality directions, unit conversion mistakes, and spurious integrality constraints. Adapting this framework to other structured generation or program repair tasks would require domain experts to manually re-engineer a new taxonomy and corresponding contract rules.

The extraction of ground-truth proxies (requirements O and instance data D) and the directional fidelity checks are performed by prompting an LLM. The paper does not quantify the error rate of this extraction stage itself or analyze how errors in O and D propagate to downstream repairs. Furthermore, using a strong frontier model (DeepSeek V4 Flash) as the repair backbone for weaker upstream generators (such as 7B/8B fine-tuned models) confounds the architectural benefits of OPT-Kraken with the raw capability gap between the generator and the verifier.

The multi-stage workflow requires multiple sequential LLM calls for extraction, directional checking, contract compilation, repair, and auditing. The experimental comparison omits compute-matched test-time scaling baselines, such as best-of-N sampling or multi-sample consistency filtering under an equal token budget.

The evaluated benchmarks consist of short, self-contained word problems with small embedded parameters, leaving open how the dual-evidence extraction and directional checks scale to realistic industrial optimization workflows where problem logic and large-scale data tables are decoupled.

Questions for authors

What is the empirical accuracy of the initial O/D extraction stage, and how does OPT-Kraken perform when the repair backbone is restricted to the exact same model as the upstream generator (e.g., repairing Qwen2.5-7B outputs using Qwen2.5-7B)?

How does OPT-Kraken compare against a compute-matched best-of-N sampling baseline that uses the same total token budget per problem?

How were the 24 RepairCard categories constructed, and were any of these rules derived by inspecting failure cases on the test benchmarks?

7 comments
AuthorsOpenReview ↗

Evidence on O/D Extraction, Same Model Repair, Sampling Baselines, and RepairCards-1

Dear Reviewer 334a:

We thank the reviewer for the thoughtful review and for recognizing the importance of semantic modeling errors and the value of evaluating rescue and regression separately. Your comments have helped us sharpen the evaluation of OPT-Kraken, particularly the reliability of extracted evidence and the benefits of structured repair relative to additional sampling. Below, we provide expert-validated O/D extraction results, a same-model 8B repair experiment, and an additional best-of-N repair comparison to address these concerns directly.

Please see below for our point-by-point responses to your comments and questions. Changes in the revised manuscript are highlighted in blue.


1. Response to empirical accuracy and downstream effects of initial O/D extraction

Reviewer comment (Question): “What is the empirical accuracy of the initial O/D extraction stage?”

Reviewer comment (Weakness): “The paper does not quantify the error rate of this extraction stage itself or analyze how errors in O and D propagate to downstream repairs.”

A1: We thank the reviewer for requesting a direct empirical assessment of the initial O/D extraction stage and its effect on subsequent repair. This helps us assess how accurately the extracted requirements and data reflect the source problem, and what happens when they are used in repair. Three experts with PhDs in mathematics and expertise in operations research and optimization manually checked the archived initial extractions against the original problems. The review covered all 801 problems for each of DeepSeek V4 Flash and Qwen3.8-27B. We have added the extraction accuracy results to Appendix C.10 (Table 17, p. 33 of the revised manuscript).

We measure extraction accuracy separately for O and D as the fraction of problems whose extraction contains no substantive error relative to the original problem. For D, we evaluate the validated data manifest and count empty extractions as errors. Every rate uses all 801 problems as its denominator. The joint O/D pass rate requires both components to pass on the same problem.

Table 1: Problem level empirical accuracy of initial O/D extraction. All rates use the full 801-problem cohort per backbone; empty D extractions count as errors.

Extraction backboneO accuracyD accuracyJoint O/D pass rate
DeepSeek V4 Flash90.0% (721/801)97.0% (777/801)87.3% (699/801)
Qwen3.8-27B87.5% (701/801)91.9% (736/801)81.3% (651/801)

O and D show high source consistency, with D more accurate for each backbone. Both extractions pass in 87.3% of the DeepSeek cases and 81.3% of the Qwen cases, providing source-consistent evidence for OPT-Kraken's subsequent diagnosis and repair.

To illustrate how errors in O and D can propagate to downstream repairs, we have added a concrete trace for MAMO Complex-112 in Appendix C.10 (p. 33 of the revised manuscript). The extraction interprets “144 grams of protein” as a carbohydrate requirement, and the repair candidate uses that interpretation in a constraint. The LLM audit returns PASS. A separate evidence check, however, finds that the candidate also changes integer variables to continuous variables without sufficient source support, so the candidate is not committed. This example traces an extraction error into a repair candidate and illustrates OPT-Kraken's separation of candidate generation and evidence-based acceptance.

AuthorsOpenReview ↗

Evidence on O/D Extraction, Same Model Repair, Sampling Baselines, and RepairCards-2


2. Response to same model repair with a weaker backbone

Reviewer comment (Weakness): “Furthermore, using a strong frontier model (DeepSeek V4 Flash) as the repair backbone for weaker upstream generators (such as 7B/8B fine-tuned models) confounds the architectural benefits of OPT-Kraken with the raw capability gap between the generator and the verifier.”

Reviewer comment (Question): “How does OPT-Kraken perform when the repair backbone is restricted to the exact same model as the upstream generator (e.g., repairing Qwen2.5-7B outputs using Qwen2.5-7B)?”

A2: We thank the reviewer for raising this concern, which helps us assess OPT-Kraken under more limited model capacity. Table 4(a) in the main paper reports same-model generation and repair results for DeepSeek V4 Flash, Qwen3.8-27B, and GPT-5.6 Sol, with each model repairing its own zero-shot outputs. These three models are relatively capable, so the weaker-backbone setting warrants a separate evaluation.

To address this concern, we selected Qwen3-8B as a representative small backbone for an additional experiment on all 801 problems. Qwen3-8B generates the initial mathematical model and code in a zero-shot setting, then repairs them within OPT-Kraken using the same checkpoint. The overall results and results for each benchmark are now included in Appendix C.11 (Tables 18 and 19, pp. 33–34 of the revised manuscript).

Table 2: Same-model generation and repair on 801 problems. Generation and repair use the identical Qwen3-8B checkpoint.

SettingAccuracyRescueRegressionError Coverage
Qwen3-8B zero-shot18.0% (144/801)———
Qwen3-8B + OPT-Kraken26.0% (208/801)10.2% (67/657)2.1% (3/144)78.5% (516/657)

OPT-Kraken increases accuracy by 8.0 percentage points, correcting 67 initially incorrect artifacts while preserving 141 of the 144 initially correct results. Rescue is lower than in the stronger-backbone settings; we discuss the effect of backbone capability in the main-text limitations (Section 5.6, p. 10 of the revised manuscript). These results show that OPT-Kraken can repair a weaker 8B model's own outputs while preserving most initially correct results.


3. Response to best-of-N sampling under comparable token budgets

Reviewer comment (Question): “How does OPT-Kraken compare against a compute-matched best-of-N sampling baseline that uses the same total token budget per problem?”

A3: We thank the reviewer for this suggestion, which helps us assess how the compute budget affects repair performance. Appendix C.6, Table 13(a) reports the average total token usage of both upstream modeling frameworks and repair baselines: 34,614 for OptiMUS-0.3, 53,127 for SAC-Opt, and 12,774 for Multi-stage Prompt-based Repair.

We understand the reviewer's concern about matching compute budgets. We therefore evaluated a best-of-N multi-sample repair baseline on the same 801 zero-shot artifacts at a similar average token budget. The baseline generates multiple complete repair candidates from the same initial model and code. A generic task-based selector reads the problem and candidate artifacts, then chooses a candidate or retains the original. We use DeepSeek V4.1 Flash for this baseline because the original V4 Flash API has been retired. DeepSeek reports stronger performance for V4.1 Flash on several agent benchmarks. The baseline uses 33,646 tokens per problem on average, compared with 34,000 for OPT-Kraken. We have added its resource usage to Appendix C.6, Table 13 (p. 31 of the revised manuscript).

Table 3: Repair results on 801 DeepSeek zero-shot artifacts at similar average token cost.

MethodRepair backboneInitial accuracyFinal accuracyRescueRegressionMean total tokens
Multi-sample repair + task-based selectionDeepSeek V4.1 Flash44.8% (359/801)60.5% (485/801)29.4% (130/442)1.1% (4/359)33,646
OPT-KrakenDeepSeek V4 Flash44.8% (359/801)64.9% (520/801)36.9% (163/442)0.6% (2/359)34,000

With similar average token usage, OPT-Kraken achieves 7.5 percentage points higher Rescue and 4.4 points higher final accuracy, with fewer regressions. Its evidence-guided diagnosis and constrained repair correct more errors and preserve more initially correct outputs than this multi-sample baseline.

AuthorsOpenReview ↗

Evidence on O/D Extraction, Same Model Repair, Sampling Baselines, and RepairCards-3


4. Response to RepairCard construction and rule provenance

Reviewer comment (Question): “How were the 24 RepairCard categories constructed, and were any of these rules derived by inspecting failure cases on the test benchmarks?”

A4: We thank the reviewer for this question, which helps us clarify the construction of the RepairCard catalog and its relationship to the test benchmarks. We first drew on the optimization modeling process to identify potential errors that LLMs might make and constructed development data covering diverse optimization tasks. The development materials included verified correct problems and modeling artifacts, erroneous cases created through controlled modifications, and modeling artifacts naturally generated by multiple LLMs. We compared the problem descriptions, mathematical models, and solver code to analyze discrepancies in objectives, variables, constraints, data mapping, and code implementation.

From this analysis, we identified 83 fine-grained modeling hallucination types and documented their definitions, typical error forms, concrete examples, and possible consequences. We then grouped and refined related types according to their causes, category boundaries, and repair needs, yielding 24 categories. For each category, we developed a RepairCard specifying its applicability conditions, required evidence, permitted and prohibited edits, and acceptance criteria. No RepairCard rules were derived from inspecting failure cases on the test benchmarks.


5. Response to RepairCard scope and cross-domain adaptation

Reviewer comment (Weakness): “The local repair mechanism relies on a manually constructed catalog of 24 OR-specific error categories (RepairCards), such as flipped inequality directions, unit conversion mistakes, and spurious integrality constraints. Adapting this framework to other structured generation or program repair tasks would require domain experts to manually re-engineer a new taxonomy and corresponding contract rules.”

A5: We thank the reviewer for raising this question about the scope and portability of RepairCards. The 24 categories describe recurring errors in the optimization modeling process, rather than errors specific to a particular application domain. They cover errors in objectives, variables, constraints, data binding, model-to-code translation, and result extraction. The same frozen catalog is used across the six optimization benchmarks and multiple upstream generators in our evaluation, without constructing a separate taxonomy for each benchmark.

RepairCards guide local repair. When source-grounded evidence indicates that local edits are insufficient, OPT-Kraken can use structural reconstruction. Eligible remaining errors can also receive a direct repair followed by an independent audit. These paths allow repair attempts for errors that may not fit a local RepairCard; success still depends on the available evidence and the model's capabilities.

Our study focuses on hallucination repair for LLM-based optimization modeling, consistent with the Optimization area of our submission. RepairCards use optimization modeling knowledge to define edit boundaries and acceptance criteria. Extending the evidence-guided repair and auditing mechanisms to other structured generation or program repair tasks is a direction for future research. In response to your suggestion, Section 6 (p. 10 of the revised manuscript) now outlines future work on using domain knowledge from areas such as power system optimization to strengthen evidence extraction and repair verification.

Reviewer_SnC4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents OPT-Kraken, a training-free framework for repairing mathematical models and solver code generated by LLMs. It extracts requirements and data from the problem statement, detects inconsistencies across artifacts, selects a repair scope, and constrains local modifications through executable contracts. Candidates undergo execution checks and semantic auditing before acceptance. The evaluation measures correction of initially incorrect outputs, corruption of initially correct outputs, and screening coverage.

The problem is important, and the reported improvements are substantial. My main reservations concern the fairness of the strongest generic baseline, incomplete evidence for semantic preservation, and the distinction from existing semantic correction methods.

Critical strengths
  • A well-motivated repair objective. Checking whether code executes or returns the expected objective cannot establish that it faithfully represents the original problem. Jointly considering problem–model fidelity, model–code consistency, and data grounding is appropriate.

  • A concrete mechanism for controlling edits. RepairCards, instance-specific RepairTasks, and contracts provide more structure than an unconstrained critique-and-rewrite prompt. Separating writable locations from read-only dependencies is particularly useful.

  • Broad and consistently positive reported results. Across eight upstream methods, the framework improves outcome accuracy. On DeepSeek zero-shot artifacts, accuracy rises from 44.8% to 64.9%, with 163 rescues and two regressions among 801 instances. The improvements across three repair backbones strengthen the empirical case.

Critical weaknesses
  • The generic multi-stage baseline does not appear to use the same acceptance policy.

In Appendix B.3, the baseline performs diagnosis, repair, review, and revisions, but ultimately returns execute(candidate) even if the final review still fails. It also generates a candidate regardless of whether the initial diagnosis identifies an error. OPT-Kraken instead retains the original artifacts unless the candidate passes its acceptance gates. This matters directly for the reported regression difference: 8.4% for Multi-stage versus 0.6% for OPT-Kraken. Some of that difference could arise from retaining failed candidates versus rolling them back, independently of directional detection, repair granularity, or contracts. The appropriate additional control is a generic multi-stage repairer with the same no-error retention, execution validation, and failed-audit rollback rules. The existing comparison establishes an advantage over the implemented baseline, but does not cleanly establish the claimed mechanism-level explanation.

Questions for authors
  1. How much of the advantage remains when Multi-stage uses identical retention and rollback rules?
  2. What is the source-level regression rate among initially faithful artifacts that receive edits?
  3. Does the advantage persist when generic repair uses the same ModelIR execution adapter?
  4. How do both local repair methods perform on SAC-Opt’s supported-target subset?
  5. For an artifact with several independent errors, can a partial correction be retained, or does the global fidelity audit reject it until all errors are fixed?
3 comments
AuthorsOpenReview ↗

Retention and Rollback, Source Fidelity, and Controlled Repair Comparisons-1

Dear Reviewer SnC4:

We thank the reviewer for the careful and constructive review. We especially appreciate the close examination of our edit-control mechanism and the recognition of the repair objective and empirical improvements. We also value the methodological expertise reflected in your questions about fair comparisons and source-level preservation. The distinction you draw between candidate generation and acceptance has been particularly valuable in refining our experimental controls. Your questions have deepened our understanding of how repair generation, acceptance policies, and source fidelity should be evaluated together.

Please see below for our point-by-point responses to your comments and questions. Changes in the revised manuscript are highlighted in blue.


1. Response to retention and rollback in Multi-stage repair

Reviewer comment (Question): “How much of the advantage remains when Multi-stage uses identical retention and rollback rules?”

A1: We thank the reviewer for highlighting the role of retention and rollback. This helps us examine how these rules affect the comparison. We replayed Multi-stage with retention and rollback and added the comparison to Appendix C.8 (Table 15, p. 32 of the revised manuscript).

Table 1: Original Multi-stage and the reviewed replay with retention and rollback.

MetricOriginal Multi-stageMulti-stage + retention/rollback
Rescue118/442 (26.7%)108/442 (24.4%)
Regression30/359 (8.4%)12/359 (3.3%)
Final Accuracy447/801 (55.8%)455/801 (56.8%)

The replay rescues 10 fewer cases but avoids 18 regressions, yielding eight more correct outputs overall. Even with retention and rollback applied to Multi-stage, OPT-Kraken achieves higher Rescue (36.9% vs. 24.4%), lower Regression (0.6% vs. 3.3%), and higher final accuracy (64.9% vs. 56.8%).


2. Response to source-level regression after edits

Reviewer comment (Question): “What is the source-level regression rate among initially faithful artifacts that receive edits?”

A2: We thank the reviewer for this question, which helps us assess whether retained edits preserve source fidelity. In the paired review of 339 artifacts, nine were initially labeled source-faithful. One was edited and remained faithful on reassessment; the other eight were retained unchanged. The source-level regression rate among edited artifacts is therefore 0/1 (0%). The code edit changed result rounding while preserving the core objective and constraints. Here, an edit means a difference between the initial artifact and the final retained model or code.

Table 2: Retained edits and source-level regression among artifacts initially labeled faithful.

MetricReassessment result
Artifacts initially labeled faithful9
Final retained model or code edited1
Final retained model and code unchanged8
Edited artifacts remaining faithful1/1
Edited artifacts with F→U0/1 (0%)

All nine initially faithful artifacts remained faithful, including the edited case, providing a concrete example of OPT-Kraken preserving source fidelity during repair.

3. Response to comparison using a shared ModelIR execution adapter

Reviewer comment (Question): “Does the advantage persist when generic repair uses the same ModelIR execution adapter?”

A3: We thank the reviewer for suggesting this control, which helps us test whether the repair gains persist with a shared execution adapter. Generic + ModelIR retains Generic’s generation inputs and audit rules, changing only the output format and deterministic compilation/execution interface. All methods start from the same 140 artifacts at the fixed repair entry: 40 initially incorrect artifacts and 100 initially correct controls. Initial accuracy is 71.4% (100/140). Because the original DeepSeek V4 Flash API has been retired, all methods in this comparison use DeepSeek V4.1 Flash. We have added the results to Appendix C.9 (Table 16, p. 32 of the revised manuscript).

Table 3: Repair performance on the 140 fixed-entry artifacts using DeepSeek V4.1 Flash.

MetricGeneric (Python output)Generic + ModelIROPT-Kraken
Rescue10.0% (4/40)12.5% (5/40)30.0% (12/40)
Regression1.0% (1/100)0% (0/100)0% (0/100)
Final Accuracy73.6% (103/140)75.0% (105/140)80.0% (112/140)

With the shared ModelIR adapter, OPT-Kraken rescues seven more artifacts (12 vs. 5) and yields seven more correct outputs (112 vs. 105), retaining its advantage in Rescue and final accuracy.

AuthorsOpenReview ↗

Retention and Rollback, Source Fidelity, and Controlled Repair Comparisons-2


4. Response to repair performance on SAC-Opt’s supported-target subset

Reviewer comment (Question): “How do both local repair methods perform on SAC-Opt’s supported-target subset?”

A4: We thank the reviewer for this careful and specific question. It helps distinguish target coverage from effectiveness on shared targets. In the original fixed-entry experiment, we compare the methods on the 17 constraint targets supported by SAC-Opt: eight initially incorrect artifacts and nine initially correct controls.

Table 4: Comparison on SAC-Opt’s supported-target subset.

MetricSAC-OptOPT-Kraken (including fallback)
Rescue1/8 (12.5%)3/8 (37.5%)
Regression2/9 (22.2%)0/9 (0%)
Final Accuracy8/17 (47.1%)12/17 (70.6%)

On this shared subset, OPT-Kraken rescues more incorrect artifacts and preserves all nine initially correct controls. Final accuracy is higher by 23.5 percentage points, showing that its advantage also holds for target types supported by SAC-Opt.


5. Response to partial correction and artifact-level audit

Reviewer comment (Question): “For an artifact with several independent errors, can a partial correction be retained, or does the global fidelity audit reject it until all errors are fixed?”

A5: We thank the reviewer for this question, which helps clarify the distinction between local repair scope and artifact-level acceptance. A Contract defines permitted edits and protected content; the artifact-level audit determines whether a candidate can replace the retained model–code pair. If a candidate corrects one error but the audit identifies another material inconsistency, the candidate is rejected and the previous pair is retained. Completing the current RepairTask alone is therefore insufficient for a commit.

When the protocol allows another refinement step, audit findings can guide further revisions to the candidate while the retained artifacts remain unchanged. The mamo_complex:182 trace illustrates this rule: a candidate corrected the opening costs and executed successfully, but the audit found fabricated transportation costs, demands, and capacities. The audit returned FAIL, so the partial correction was not committed.

The acceptance and retention rules are described in Section 4.5 and Appendix B.4. We have added this case to Appendix C.13 (p. 35 of the revised manuscript).


We thank you again for your careful review and constructive comments, which have helped us improve this work.

Best regards,
Authors

Thank you for the answers to my questions and improving the paper. I will keep my rating, namely weak ACCEPTANCE.

Reviewer_K1p9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on a setup where a mathematical model and solver code is generated for a high level optimization problem expressed in natural language. The main ideas are around identifying the scope of problem/changes and fixing it in a structured manner. Comparison against prompt based baselines shows positive results. In terms of modeling novelty and complexity, the paper is on the limited side. The paper is also a bit difficult to follow and the writing can improve (the text is not coherent, and paragraphs feel like a list of requirements/propositions in many parts of the paper).

Critical strengths

The main novelty of the paper is focusing on hallucinations specific to optimization setups, and fixing them in a scoped and structured manner. The comparisons against prompt based baselines show positive lifts.

Critical weaknesses

The scope of the setup is limited to hallucinations in the context of optimization problems. Secondly, while the proposed approach is intuitive the modeling novelty/complexity is limited, and the baselines are limited to prompt based approaches (which are also simple) - its not clear if an agent driven iterative approach would be comparable/better.
There is no discussion around the possible failure modes as well.

Questions for authors

No major questions, except for the concerns around modeling novelty, baselines, and failure modes as outlined in the weaknesses section.

4 comments
AuthorsOpenReview ↗

Clarifying the Optimization Scope and Novelty with Agent Baseline and Failure Mode Analyses-1

Dear Reviewer K1p9:

Thank you for your constructive review. Your comments have helped us clarify the scope and contributions of our work and prompted us to revisit the paper's organization and presentation. In the revised manuscript, Section 3 presents the problem formulation, and Section 4.1 and Algorithm 1 provide an overview of the framework and its decisions (pp. 3–4 of the revised manuscript). The problem formulation explains how optimization modeling, upstream generation, and repair are related. Algorithm 1 follows the process from diagnosis and repair-scope selection to candidate auditing. Sections 4.3 and 4.4 now explain how evidence guides the repair scope and how local edits preserve unaffected content (pp. 5–6 of the revised manuscript). We have also added a discussion of failure modes and limitations in the main text and an appendix case showing how the audit handles a partial correction (pp. 10 and 35 of the revised manuscript, respectively).

Please see below for our point-by-point responses to your comments and questions. Changes in the revised manuscript are highlighted in blue.


1. Response to research scope and generality

Reviewer comment (Weakness): “The scope of the setup is limited to hallucinations in the context of optimization problems.”

A1: We thank the reviewer for raising the question of scope. Our work is intentionally focused on hallucinations in optimization modeling, consistent with the paper’s submission to the Optimization area. This setting presents a specific repair challenge: the mathematical formulation and solver code must jointly represent the requirements and data in the original problem. Addressing an error may therefore require coordinated changes across the model and code, while preserving the parts that are already correct.

Within this setting, OPT-Kraken addresses a range of modeling errors across optimization tasks. We evaluate it on six benchmarks and outputs from multiple upstream generators. Extending its evidence-guided diagnosis and constrained repair to other mathematical modeling tasks is a direction for future research.


2. Response to methodological novelty and technical complexity

Reviewer comment (Weakness): “Secondly, while the proposed approach is intuitive the modeling novelty/complexity is limited.”

A2: Thank you for prompting us to clarify the contribution. To the best of our knowledge, OPT-Kraken is the first post-hoc framework for repairing hallucinations in LLM-generated optimization-modeling artifacts. We formulate repair as a standalone task on existing model–code pairs, with two objectives: correcting errors and preserving initially correct artifacts. Rescue and Regression measure these objectives separately, while Error Coverage measures how many initially incorrect artifacts are identified by screening. This distinguishes successful corrections from harmful edits that aggregate accuracy can obscure.

For readability, the overview focuses on the core mechanisms. Sections 4.3–4.5 (pp. 5–7 of the revised manuscript) give the detailed decision and validation rules. The technical complexity lies in turning a localized error into a coordinated model–code correction while preserving unaffected content. OPT-Kraken uses the error location and its dependencies to determine the repair scope, then compiles category-level RepairCards and instance-specific RepairTasks into executable edit restrictions and acceptance conditions. These distinguish writable locations from read-only dependencies and specify what to correct and preserve. Candidate generation and acceptance are separate decisions: a proposed repair must satisfy the edit constraints and pass the artifact-level audit before replacing the retained artifacts.

The mechanism ablations in Table 5 support these design choices: Rescue is 36.9% for the full system, versus 12.4% without dual-evidence diagnosis and 24.4% without contracted local repair.

AuthorsOpenReview ↗

Clarifying the Optimization Scope and Novelty with Agent Baseline and Failure Mode Analyses-2


3. Response to agent-driven iterative baselines

Reviewer comment (Weakness): “And the baselines are limited to prompt based approaches (which are also simple) - its not clear if an agent driven iterative approach would be comparable/better.”

A3: We appreciate the suggestion to compare against an iterative agent. Our existing Multi-stage Prompt baseline already follows an agent-style iterative diagnosis–repair–review workflow, with revision after a failed review (Appendix B.3); our description did not make this sufficiently clear. We have also evaluated an adaptation inspired by Reflexion (Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao, “Reflexion: Language Agents with Verbal Reinforcement Learning,” Advances in Neural Information Processing Systems 36, pp. 8634–8652, 2023). It uses nine stages of candidate generation, textual review, reflection, revision, and final selection, with prior feedback retained as context.

We evaluated the method on all 801 frozen DeepSeek V4 Flash Zero-shot artifacts. Because the original V4 Flash API has been retired, the new baseline uses DeepSeek V4.1 Flash; DeepSeek reports stronger V4.1 results on several reasoning and agent benchmarks. We have added this comparison to Appendix C.7 (Table 14, p. 32 of the revised manuscript) and its resource usage to Table 13(a) (p. 31 of the revised manuscript). The results are:

Table 1: Comparison on 801 DeepSeek V4 Flash Zero-shot artifacts.

Repair methodInitial AccuracyFinal AccuracyRescueRegressionError CoverageMean total tokensMean logical calls
Reflexion-inspired, nine-stage (V4.1 Flash)44.8% (359/801)61.0% (489/801)31.7% (140/442)2.8% (10/359)N/A37,0669.00
OPT-Kraken (V4 Flash)44.8% (359/801)64.9% (520/801)36.9% (163/442)0.6% (2/359)94.1% (416/442)34,0003.65

Error Coverage is N/A for the Reflexion-inspired baseline because it does not produce a comparable initial screening decision. OPT-Kraken rescues 23 more artifacts and introduces 8 fewer regressions, using fewer logical calls and fewer total tokens on average. Rescue is higher by 5.2 percentage points and final accuracy by 3.9 points, while Regression is lower by 2.2 points. OPT-Kraken thus corrects more errors and preserves more correct outputs than this iterative baseline, at a lower average inference cost.


4. Response to failure modes and limitations

Reviewer comment (Weakness): “There is no discussion around the possible failure modes as well.”

A4: Thank you for suggesting a discussion of failure modes. This helps us clarify when repair may fail and the limitations of the auditing mechanism. We have added a Failure Modes and Limitations subsection in Section 5.6 (p. 10 of the revised manuscript). It explains that missed errors, insufficient evidence, or failed audits can leave incorrect artifacts unrepaired, and that harmful edits can pass the audit. On the DeepSeek V4 Flash zero-shot outputs, 279 of 442 initially outcome-incorrect artifacts remain incorrect, while 2 of 359 initially outcome-correct artifacts became incorrect. In Appendix C.13 (p. 35 of the revised manuscript), we added an illustrative case in which a partial correction executes successfully but is rejected because material inconsistencies remain. The original artifacts are therefore retained.


We thank you again for your careful review and constructive comments, which have helped us improve this work.

Best regards,
Authors

AuthorsOpenReview ↗

Follow up on Our Rebuttal

Dear Reviewer K1p9:

Thank you again for your careful review and constructive feedback. We would like to follow up on our rebuttal, in which we addressed your comments and provided additional experimental results and clarifications.

As the discussion period draws to a close, we would be grateful if you could review our responses when your schedule permits. Please let us know if you have any remaining questions or concerns; we would be happy to provide further clarification.

Thank you for your time and consideration.

Best regards, Authors

AuthorsOpenReview ↗

Brief Follow-up on Our Response

Dear Reviewer K1p9:

Thank you again for your review. For response, we clarified why our work focuses on optimization modeling and explained its technical contributions. We also revised the presentation, added a discussion of failure modes, and evaluated a Reflexion-inspired baseline baseline: OPT-Kraken achieved higher rescue(36.9%) compared with 31.7%, with fewer regressions and lower average token usage.

We already upload the revised version and Changes in the revised manuscript are highlighted in blue. Since the rebuttal ends today, we would appreciate it if you could take a look at our response. Please let us know if there is anything we can clarify.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.