acceptodds
Under review as a conference paper at ICLR 2027

EffiOR: Post-Model Refinement for Efficient Solving

Abstract

Large language models (LLMs) have made substantial progress in automatically generating mathematical models and solver implementations for operations research problems. However, correct modeling does not necessarily imply efficient solving, as the same optimization problem may admit multiple valid formulations with substantially different computational performance. We propose EffiOR, a framework for improving existing optimization models and solver implementations rather than rebuilding them from scratch. EffiOR formalizes 29 reusable Optimization Skills across Model-Structure, Formulation-Refinement, and Solver-Implementation, enabling existing models and solver implementations to be transformed for more efficient solving while preserving the original problem semantics. Experiments on FrontierOR-179 show consistent efficiency gains, while four additional optimization-modeling benchmarks demonstrate high transformation consistency.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Models, Optimization Modeling, Efficient Solving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5BCHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EffiOR improves an existing operations-research formulation and solver program rather than generating a model from scratch. It aligns the problem description, formulation, and code through a semantic intermediate representation, selects among 29 established optimization skills using 39 shared selection features, and updates the representations after each transformation. Instrumented solver runs support objective and stricter solver-state consistency checks. Experiments combine two upstream modeling frameworks with three refinement backbones on FrontierOR-179, plus consistency evaluations on four smaller modeling benchmarks. EffiOR outperforms a generation-budget-matched Iterative CoT baseline in all six reported FrontierOR settings. It improves average QTE over the original models by 14.24 percentage points, although quality gains vary across settings.

Critical strengths
  1. The paper identifies a useful downstream task and provides a concrete implementation of established OR knowledge for LLM agents. The skill catalog includes applicability conditions and formulation/code examples, while the evolving IR records how solutions map back to the original model. Shared necessary-condition screening is a sensible way to reduce repeated applicability reasoning. The contribution is the organized application and checking of these transformations, rather than a claim to have invented the underlying OR techniques.
  2. The evaluation separates upstream modeling from refinement by freezing the input artifacts, and includes a same-backbone, generation-budget-matched baseline. The strongest setting improves OptiMUS QTE from 35.75% to 56.42%, and the skill-category ablation locates most of the gain in formulation refinement. Reporting stricter state consistency alongside objective agreement is valuable: the 94.47% versus 88.73% macro averages on the smaller benchmarks expose discrepancies that objective agreement alone would hide. The manuscript also acknowledges quality regressions for some LLMOPT/backbone combinations.
Critical weaknesses

The efficiency and consistency results are reported separately, leaving the usable benefit of verified transformations unclear. On FrontierOR, strict consistency ranges from 83.80% to 95.53%, so a nontrivial fraction of outputs fail the paper's stronger check. The paper does not clearly report the intersection of QTE improvements with successful consistency checks, or performance after falling back to the original artifact when a check fails. Moreover, comparing objectives within 1% and matching solver status can flag a valid reformulation that finds a better incumbent or proves optimality within the time limit. Conversely, agreement on a finite execution does not establish general semantic equivalence. A failure breakdown is needed to distinguish semantic errors from conservative rejections and valid improvements.

Questions for authors
  1. Can you report, per setting, QTE and quality after rejecting inconsistent transformations and reverting to the frozen baseline? Please also report how many efficiency gains pass the strict check and break down failures into semantic violations, execution failures, and valid changes in objective or solver status.
  2. How does the verifier handle an exact reformulation that improves a time-limited incumbent by more than 1%, changes TIME_LIMIT to OPTIMAL, or replaces one solve with a decomposition requiring several solves? Please state which checks compare semantic outcomes and which require identical execution states or sequences.
  3. Can a matched-budget control use the same 29 skills but check applicability directly, without the shared-feature routing layer? An IR ablation or routing-cost measurements would clarify which part of the proposed mechanism contributes beyond supplying OR knowledge.
  4. What are the actual generation budgets, number of solver executions, and refinement/verification costs for EffiOR and Iterative CoT? Please distinguish final-program runtime from total one-off refinement cost and discuss when repeated use amortizes that cost.
3 comments

We sincerely thank the reviewer for the careful evaluation and constructive feedback.

Weaknesses:

Thank you for pointing out the need to better explain the relationship between solving efficiency and transformation consistency. In our experiments, we use a strict evaluation protocol in which State Consistency is a prerequisite for counting a transformed result as successful in the solution-quality and efficiency metrics. Specifically, for each instance in FrontierOR-179, we first perform strict consistency verification between the original and transformed programs. Only transformations that pass the State Consistency check are counted toward Qual@2%, Qual@1%, and QTE. If a transformation fails this check, the instance is counted as unsuccessful, even if its objective value or runtime appears better. These instances remain in the original denominator of 179 rather than being removed from the evaluation. Therefore, our reported solution-quality and efficiency results have already undergone strict consistency filtering. For example, with OptiMUS and GPT-5.6 Sol, 171 out of 179 instances pass State Consistency, among which 101 also satisfy the solution-quality and runtime requirements of QTE. Thus, the reported QTE of 56.42% includes only results that have passed strict consistency verification. If QTE were calculated only over instances that pass State Consistency, excluding the failed instances from the denominator, the resulting conditional QTE rate would be higher. For the above setting, it would be 101/171 = 59.06%, compared with 56.42% when all 179 instances are included. We retain all 179 instances in the denominator to avoid inflating the overall performance by excluding cases that fail verification. When a consistency check fails during transformation, the agent further examines its internal transformations, locates the inconsistent variables, constraints, or objective expressions, and attempts to repair them before performing verification again. Results that still fail the final strict check are not counted as performance improvements. Thus, our reported QTE and solution-quality metrics reflect solving improvements that pass strict consistency verification over the complete test set, rather than potential gains from unverified transformations.

Question 1:

Thank you for pointing out the need to better explain the relationship between solving efficiency and transformation consistency. In our evaluation, strict State Consistency is a prerequisite for counting a transformed result as successful in the solution-quality and efficiency metrics. Specifically, if a transformed result fails the strict consistency check, it is counted as unsuccessful, regardless of its observed objective value or runtime. Such instances remain in the original denominator of 179 rather than being removed from the evaluation. Therefore, our reported solution-quality and efficiency results have already undergone strict consistency filtering. For example, with OptiMUS and GPT-5.6 Sol, 171 out of 179 instances pass State Consistency, among which 101 also satisfy the solution-quality and runtime requirements of QTE. Thus, the reported QTE of 56.42% includes only results that have passed strict consistency verification. The same rule applies to all six framework–backbone settings. If QTE were calculated only over instances that pass State Consistency, excluding the failed instances from the denominator, the resulting conditional QTE rate would be higher. For the above setting, it would be 101/171 = 59.06%, compared with 56.42% when all 179 instances are included. We retain all 179 instances in the denominator to avoid inflating the overall performance by excluding cases that fail verification. When a consistency check fails during transformation, the agent further examines its internal transformations, locates the inconsistent variables, constraints, or objective expressions, and attempts to repair them before performing verification again. Results that still fail the final strict check are not counted as performance improvements. In this way, an improvement is counted only when it passes both the consistency check and the corresponding quality or runtime requirement.

Question 2:

We use small-scale instances to examine whether the transformations preserve the original mathematical problem. These instances are chosen so that both the original and transformed formulations can be solved to proven optimality within the available budget. For each instance, we examine the correspondence maintained by the IR, including the mappings of variables, constraints, and objective terms. We then map the transformed solution back to the original variable space, check its feasibility against the original constraints, and compare the optimal objective values. We do not regard agreement of optimal objective values alone as sufficient. It must be accompanied by a valid IR mapping and feasibility preservation. For transformations involving auxiliary variables or variable elimination, we check the corresponding solution-recovery or projection mapping rather than requiring a one-to-one correspondence between all variables. These small-instance checks provide a direct way to examine mathematical preservation without the interference of time-limited solver outcomes, since both formulations are solved to proven optimality. For larger instances, a better incumbent or a change from TIME_LIMIT to OPTIMAL does not by itself imply an invalid reformulation. Such cases need to be assessed through the recovered solution, original-space feasibility, objective evaluation, and, where optimality is claimed, the corresponding bound or optimality evidence. Our current strict State Consistency criterion also checks solver states and solve sequences. It is therefore a conservative execution-level measure. A transformation that changes the number of solver calls or the solver status may fail this strict check even when its mathematical formulation is valid. We distinguish this execution-level criterion from mathematical equivalence.

Question 3:

We conducted an additional ablation to examine the contribution of the shared feature-screening mechanism. Direct Skill Checking retains the same IR and all 29 Optimization Skills, but removes the 39 shared Selection Features and their associated Skill Gates. Instead, it directly checks the original 117 Skill-specific applicability conditions. The results show a clear difference. Direct Skill Checking achieves 38.55% QTE, while full EffiOR achieves 56.42%, an increase of 17.87 percentage points. Qual@2% and Qual@1% also increase from 60.34% and 54.19% to 75.98% and 69.83%, respectively. Objective and state consistency improve from 93.85% to 97.33% and from 89.94% to 95.53%. The 39 shared Features are not replacements for the original applicability conditions. They capture necessary structural conditions shared by different Skills and are used for preliminary screening. A Skill that passes its gate still undergoes its complete applicability check.These results suggest that the feature-screening and Skill-gating mechanism contributes beyond simply providing the same OR knowledge. We acknowledge that this ablation retains the IR and therefore does not independently measure the contribution of IR alignment.

MethodQual@2% ↑Qual@1% ↑QTE ↑Obj. Cons. ↑State Cons. ↑
EffiOR (Full)75.9869.8356.4297.3395.53
Refinement + Solver73.1867.6054.7597.3395.53
Solver Only61.4554.1935.2093.1789.39
Direct Skill Checking60.3454.1938.5593.8589.94
Original62.5754.1935.75——

Question 4:

We measured the refinement cost of EffiOR by stage, including LLM calls, token consumption, and wall-clock time.The four stages are standardized structural extraction, feature evaluation and Skill invocation, ordered Skill application, and consistency verification. Their total cost is 17.17 LLM calls, 70.33K tokens, and 445.83 seconds per task. For comparison, Iterative CoT requires an average of 12.50 calls, 41.83K tokens, and 301.67 seconds. Thus, EffiOR has a higher one-off refinement cost. The break-even point therefore depends on the actual runtime reduction and the number of subsequent executions.We distinguish LLM calls from solver executions, since they measure different operations. Solver-execution counts and final-program runtime are reported separately from the one-off refinement cost.

Refinement StageLLM CallsTotal TokensWall-clock (s)
1. Standardized Structural Extraction3.43414.77K115.916
2. Refined Feature Evaluation and Skill Invocation1.7176.33K35.666
3. Ordered Skill Application8.24234.46K182.790
4. Consistency Verification3.77714.77K111.458
Total Refinement17.17070.33K445.830
Reviewer_3u7yOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EffiOR addresses post-model refinement: given an existing problem description, formulation, and solver implementation, it applies optimization transformations to improve computational performance. The paper organizes 29 optimization skills into three categories and uses a semantic IR and 39 shared selection features to support skill selection and application. Experiments cover two upstream modeling frameworks and three backbone LLMs.

Critical strengths
  1. The paper addresses a practically important problem: a correct and executable optimization model can still be computationally inefficient, leaving substantial opportunities for post-model refinement.
  2. A valuable contribution is the detailed organization of the optimization strategy space. The 29 skills span model structure, formulation refinement, and solver implementation, with applicability conditions, mathematical explanations, and examples. This taxonomy provides a concrete foundation for making established OR expertise accessible to modeling agents.
  3. The framework connects problem semantics, mathematical formulations, and executable implementations through an IR, supporting coordinated transformations and solution recovery across representations.
  4. Evaluation across multiple upstream systems and LLMs, skill-category ablations, and before-and-after cases provides useful evidence of the approach’s potential and practical behavior.
Critical weaknesses
  1. The net computational benefit is not established. IR construction, feature evaluation, applicability checking, transformation generation, and validation introduce additional costs. The solving metrics do not establish whether downstream savings outweigh this overhead. Generation-budget matching against Iterative CoT does not replace end-to-end cost accounting against the original pipeline. Token usage and construction, validation, and execution time should be reported separately, with amortization analysis where repeated reuse is intended.
  2. Correctness and efficiency are not sufficiently evaluated jointly. Quality, timing, and transformation consistency are reported separately, leaving unclear how often an individual transformation is both valid and beneficial. Execution-level consistency is also narrower than formulation equivalence. Per-instance joint success rates requiring original-space feasibility, qualified solution quality, and the timing criterion would provide stronger evidence. The treatment of consistency failures and legitimate solver-status changes should be clarified.
  3. The comparison with skill-equipped agents is limited. Iterative CoT is an informative reasoning baseline, but does not represent a coding agent equipped with independently developed OR modeling and optimization skills. Such a comparison would better contextualize EffiOR’s contribution relative to existing skill-based workflows. Separately, internal ablations removing IR alignment or feature screening while retaining the proposed skill library would help identify the contribution of each architectural component.
  4. The screening mechanism and performance regressions need further analysis. The consolidation of 117 applicability conditions into 39 shared features is described, but screening accuracy, missed applicable skills, and reasoning-cost savings are not sufficiently evaluated. Mathematical applicability does not guarantee acceleration: tighter formulations may increase construction or per-node solving costs. Concrete regression cases and the policy for detecting and handling them are insufficiently analyzed.
Questions for authors
  1. What are the additional token usage and wall-clock costs of each refinement stage? After including these costs, when does EffiOR outperform the original pipeline, and how many repeated solves are needed to amortize refinement?
  2. Can the authors report per-instance joint quality–feasibility–time success rates and clarify whether consistency failures are rejected, counted as failures, or replaced by the original artifacts?
  3. How does EffiOR compare with a coding agent equipped with independently developed third-party OR skills under comparable resources? Can internal ablations separately assess IR alignment and feature screening?
  4. How were the 117 conditions consolidated into 39 features, and what are the screening mechanism’s false-negative rate and cost savings?
  5. Did any optimality-preserving tightening or symmetry reduction increase runtime? Please provide a representative case and explain how EffiOR handles such regressions.
  6. As an alternative pipeline, what accuracy and total cost are obtained by constructing IR directly from the description and then performing skill-guided model/code generation?
5 comments

We sincerely thank the reviewer for the careful evaluation and constructive feedback.

Weakness 1 and Questions 1:

EffiOR introduces additional costs, which we explicitly measure. Our setting is post-model refinement: the upstream formulation and solver implementation are first generated and frozen, after which EffiOR refines them offline. The resulting program can then be reused without further LLM calls. We report the LLM refinement costs across two upstream frameworks and three backbones.

FrameworkBackboneMethodAvg. LLM CallsAvg. Total TokensAvg. Refinement Time (s)
OptiMUSGPT-5.6 SolIterative CoT1241K290
OptiMUSGPT-5.6 SolEffiOR1666K420
OptiMUSGLM-5.2Iterative CoT1446K325
OptiMUSGLM-5.2EffiOR1979K485
OptiMUSQwen3.7-MaxIterative CoT1136K270
OptiMUSQwen3.7-MaxEffiOR1558K390
LLMOPTGPT-5.6 SolIterative CoT1552K360
LLMOPTGPT-5.6 SolEffiOR2191K550
LLMOPTGLM-5.2Iterative CoT1344K310
LLMOPTGLM-5.2EffiOR1874K460
LLMOPTQwen3.7-MaxIterative CoT1032K255
LLMOPTQwen3.7-MaxEffiOR1454K370
Average—Iterative CoT12.5041.83K301.67
Average—EffiOR17.1770.33K445.83

The benefit of EffiOR depends on whether the refined program is executed repeatedly.

Weakness 2:

We would like to clarify the scope of EffiOR. EffiOR is a post-model refinement component in the optimization pipeline, rather than an end-to-end modeling system. Its objective is to improve the solving efficiency of an existing mathematical formulation and solver implementation. An important distinction is between the correctness of the original model and the consistency of the refinement process. Whether the original formulation correctly represents the underlying real-world problem belongs to the upstream modeling stage and is outside the scope of EffiOR. We take the supplied formulation and solver implementation as the input to be refined, rather than attempting to verify or correct their modeling assumptions. What EffiOR is required to ensure is that its transformations preserve the semantics of the supplied model and implementation. In other words, the refined program should remain consistent with the original one, even though its internal formulation or implementation may change. We check this by mapping the refined solutions back to the original variable space, verifying the original constraints, and recomputing the objective values. A candidate that fails these checks is not accepted as a valid refinement.

Therefore, consistency is a necessary constraint on the refinement process, whereas solving efficiency is the actual optimization objective. Our goal can be summarized as improving solving efficiency subject to transformation consistency:

max⁡ SolvingEfficiency\max \ \mathrm{SolvingEfficiency}

\mathrm{TransformationConsistency}=\mathrm{TRUE}.$$ We do not aim to establish whether the original model is mathematically correct with respect to the real-world problem. If an error already exists in the supplied model, preserving its semantics does not necessarily eliminate that error. This is an intentional boundary of our work: EffiOR improves how an existing model is solved, rather than whether the original problem has been modeled correctly.

Weakness 3:

Related systems such as OptiMUS and OptDachshund mainly address the generation of mathematical formulations and solver implementations from problem descriptions. EffiOR starts from a different point: the formulation and solver code already exist, and the task is to refine these artifacts to improve solving efficiency.

We have not conducted a direct comparison with a third-party agent under the same frozen-artifact input setting and post-model refinement objective. We therefore do not claim empirical superiority over these systems. Instead, we use the internal ablation to examine the contribution of our feature-screening mechanism.

In this ablation, Direct Skill Checking retains the same IR and all 29 Optimization Skills, but removes the 39 shared Selection Features and their corresponding Skill Gates. It directly evaluates the original 117 applicability conditions.

The results are shown in Table 3.

Table 3. Ablation Study on FrontierOR-179

Framework: OptiMUS; Backbone: GPT-5.6 Sol (High). All metrics are percentages.

MethodQual@2% ↑Qual@1% ↑QTE ↑Obj. Cons. ↑State Cons. ↑
EffiOR (Full)75.9869.8356.4297.3395.53
Refinement + Solver73.1867.6054.7597.3395.53
Solver Only61.4554.1935.2093.1789.39
Direct Skill Checking60.3454.1938.5593.8589.94
Original62.5754.1935.75——

With the same IR and the same 29 skills, removing feature screening reduces QTE from 56.42% to 38.55%, a decrease of 17.87 percentage points. Objective Consistency also decreases from 97.33% to 93.85%, while State Consistency decreases from 95.53% to 89.94%.The results show that the skill library alone does not achieve the performance of the full framework. Feature screening makes the selection of optimization skills more targeted and contributes to the solving results and transformation consistency.

Weakness 4: The 117 applicability conditions are not simply replaced by 39 simplified checks. We extract recurring structural predicates from the original conditions of the 29 Optimization Skills and organize them into 39 reusable Selection Features: 9 Model-Structure Features, 13 Formulation-Refinement Features, and 17 Solver-Implementation Features.

Each skill has a gate defined by the conjunction of its required features. The gate is used for preliminary screening, while the original skill-specific conditions are still checked before the skill is applied.

The screening process is sparse. EffiOR does not evaluate all 39 features for every candidate. Once a required feature is evaluated as FALSE, the corresponding gate cannot be satisfied, so the remaining evaluations for that gate can be skipped. Only surviving skills proceed to the full applicability check. An UNKNOWN result is not used to exclude a candidate.

We illustrate this with the Bin Packing example in Figure 3. Consider a problem with bin capacity W=10W=10, five items of size 3, and three items of size 4. The objective is to minimize the number of bins. For the State-space Arc-flow skill, the IR identifies three required features:

MG2: Generating-object family — TRUE.

MG3: Complete object semantics — TRUE.

MG9: Exact finite state paths — TRUE.

Therefore, its gate is satisfied:

MG2∧MG3∧MG9MG2 \land MG3 \land MG9

Arc-flow passes the screening stage and satisfies the subsequent full applicability check. In contrast, Benders decomposition requires the master-conditional recourse structure represented by MG4, which is absent from the current IR. It is therefore not selected for this instance. This does not mean that Benders is unsuitable for all Bin Packing formulations.

The results include both improvements and regressions. For example, valid-inequality strengthening improves the objective from 74 to 34 on letelier2022, and from 1196 to 1188 on kreter2018. However, the order-cut transformation on martin-iradi2024 worsens the objective from 135653059.37 to 147515461.98 under the tested time budget. Bound tightening on reinhardt2016 also produces a worse objective, changing it from 1647.2 to 1997.7. These cases show why structural applicability and computational improvement should be treated separately. The screening mechanism determines whether a skill is supported by the identified model structure; its actual effect on solving performance still needs to be evaluated. Consistency failures are handled through diagnosis and repair before a transformed program is released. Finally, we have not separately measured the false-negative rate of feature screening or the token savings attributable specifically to the 39 shared features. We will make these limitations clear rather than infer such results from the overall ablation or refinement-cost measurements.

question 2:

When a consistency check fails, EffiOR uses the maintained IR to locate the source of the mismatch. Since the IR keeps track of the correspondence between variables, constraints, and objectives across the formulation and solver implementation, the agent can identify which part of the transformation causes the inconsistency. It then performs a targeted repair on that part and runs the consistency check again. This process is repeated until the required consistency checks are passed. In this way, EffiOR does not simply reject an entire transformation when an inconsistency is detected, but uses the IR to identify and correct the specific problem.

question 3:

We first compare the resources required by EffiOR and Iterative CoT. As reported in the refinement cost table under Question 1, the comparison covers two upstream frameworks and three backbones, using LLM calls, total tokens, and refinement time as the main resource measures. Regarding third-party agents, we distinguish between optimization modeling and post-model refinement. Existing systems such as OptiMUS mainly generate mathematical formulations and solver implementations from natural-language descriptions. More recent work, including OptDachshund, also incorporates optimization techniques to improve the computational efficiency of generated models. However, these systems start from problem descriptions and construct or reformulate models and code. They do not use the same setting as EffiOR, where an existing formulation and solver implementation are frozen before offline refinement.

We therefore have not identified a directly comparable third-party agent that satisfies both requirements: starting from an existing optimization model and solver program, and performing offline post-model refinement to improve solving efficiency. We use Iterative CoT as the closest available baseline under our current experimental setting, rather than claiming that it represents all skill-equipped agents.For internal ablations, we report the Direct Skill Checking results in Table 3. This variant retains the same IR and all 29 Optimization Skills but removes the 39 shared Selection Features and their corresponding Skill Gates. QTE decreases from 56.42% to 38.55%, while Objective Consistency decreases from 97.33% to 93.85% and State Consistency decreases from 95.53% to 89.94%.This comparison shows the contribution of feature screening. Since the IR is retained, it does not independently measure the contribution of IR alignment. We acknowledge this limitation and do not claim that the current ablations isolate every component.

question 4:

The 117 applicability conditions are not replaced by 39 simplified checks. Instead, we extract recurring necessary structural predicates from the original conditions of the 29 Optimization Skills and organize them into 39 reusable Selection Features, including 9 Model-Structure Features, 13 Formulation-Refinement Features, and 17 Solver-Implementation Features.Each Skill has a gate defined by the conjunction of its required features. These gates are only used for preliminary screening. The original Skill-specific applicability conditions are still checked before a Skill is applied. The screening process is also sparse: once a required feature is evaluated as FALSE, the corresponding gate cannot be satisfied, and the remaining evaluations exclusive to that gate can be skipped. An UNKNOWN result does not exclude a candidate.

We illustrate this with the Bin Packing example in Figure 3. Consider a problem with bin capacity (W=10), five items of size 3, and three items of size 4. For the State-space Arc-flow Skill, the current IR identifies three required features: MG2 (Generating-object family), MG3 (Complete object semantics), and MG9 (Exact finite state paths). All three are satisfied, so Arc-flow passes the initial screening and proceeds to the full applicability check. In contrast, Benders decomposition requires the master-conditional recourse structure represented by MG4, which is absent from the current IR. It is therefore not selected for this particular formulation. This does not mean that Benders is unsuitable for all Bin Packing formulations.

Regarding the screening accuracy and cost savings, we have not separately measured the false-negative rate or the token savings attributable specifically to the 39 shared features. The Direct Skill Checking ablation provides evidence of the contribution of feature screening to the overall framework, but its performance difference cannot be used to calculate the screening false-negative rate.

Reviewer_1snYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

LLMs are increasingly used to generate optimization models and corresponding solver code from natural language description. However, most existing studies only focus on generating correct models and code while overlooking the efficiency of generated models and code. This paper studies this important but under-explored topic. To this end, they propose a framework named EffiOR to improve the efficiency of already-built math models and solver code (not from scratch). Specifically, they develop 29 modeling and coding skills to achieve this, and experiments on the FrontierOR benchmark and other 4 benchmarks show promising results.

Critical strengths
  1. They study an important but under-explored topic in LLM-based optimization modeling community;

  2. The writing is smooth and very easy to follow;

Critical weaknesses
  1. [How to distinguish "wrong models" from "correct but inefficient models"?] EffiOR works under such a setting: "given an existing mathematical formulation and solver implementation, identify and apply suitable transformations that improve solving efficiency while preserving the specified problem semantics." However, how to make sure that the given mathematical formulation and solver implementation were correct (namely they preserve the original problem semantics)? What if the original mathematical formulation and/or solver implementation were wrong (rather than "correct but not efficient")? Can your framework distinguish these differences and still work?

  2. [The cost of EffiOR was not reported]: The ultimate goal of EffiOR (or other studies on improving the efficiency of optimization modeling) is to reduce the total time of model/code generation and solver execution. This study employs this additional framework to reduce the running time of solver code. However, the time cost of this process (namely the process of improving) was not reported. Consider an example as follows: the original solver code takes 10 minutes to execute, the improved solver code takes 5 minutes to execute, but the EffiOR takes 10 minutes to execute? (10 minutes without EffiOR vs 15 minutes without EffiOR)? Importantly, EffiOR works as follows: "EffiOR first constructs a semantic intermediate representation IR that aligns the problem description, mathematical formulation, and solver implementation at the variable, constraint, and objective levels." This process can be very costly when there are millions of variables/constraints.

  3. [Only two optimisation modeling systems were evaluated]: The authors only evaluated two optimisation modeling systems (two modeling frameworks and three backbone LLMs) while overlooking the performance on 1) generic-purpose LLMs such as GPT (GPT-6.0 Sol), Geimini (Gemini 3.1-pro), Claude (Sonnet 5.5), Grok (Grok $.7), DeepSeek (DeekSeek V4.1), Qwen (Qwen 3.8 max), Kimi (Kimi K3.1), etc; 2) Fine-tuned optimization modeling systems such as ORLM, OptMATH, SIRL, StepORLM, SAGE...etc. Therefore, the findings and conclusions may not generalizable.

  4. [The evaluation may be flawed]. The authors indicated that "All programs are evaluated under the same isolated environment using Python 3.13 and Gurobi 12, with four CPU cores available to each execution, identical memory and time limits, and disabled network access. The four-core limit specifies the maximum CPU resources available to a single program execution and does not force the solver to use exactly four threads". As they admit that " The four-core limit specifies the maximum CPU resources available to a single program execution and does not force the solver to use exactly four threads", therefore, some implementations that utilise four-core parallel computation can be faster (but in a very trivial sense) than implementations that use single core. For fair comparisons, these execution should be done on single-core limit.

Questions for authors

Please check the weak points.

2 comments

We thank the reviewer for the constructive comments and for recognizing the importance of this problem.

W1. Distinguishing incorrect models from correct but inefficient models.

EffiOR starts from an existing solver program and aims to improve it without changing the model it represents. When the mathematical formulation and problem description are available, we use them as additional context for constructing the IR. Otherwise, the LLM recovers the model structure directly from the solver code. Our focus is on the correctness of the transformations, rather than validating whether the original formulation accurately represents the real-world problem. An error already present in the input model may therefore remain after refinement. To check whether a transformation preserves the original model, we use small instances for which both the original and transformed programs can be solved to proven optimality. We compare their optimal objective values instead of time-limited incumbent values. We also map solutions from the transformed model back to the original variable space, check the original constraints, and recompute the objective. Large instances are evaluated separately to measure solving efficiency.

W2. Additional computational cost of EffiOR.

We now report the refinement cost in terms of LLM calls, total token consumption, and elapsed time. The results cover Iterative CoT and EffiOR under two upstream frameworks and three backbones.

FrameworkBackboneMethodAvg. LLM CallsAvg. Total TokensAvg. Refinement Time (s)
OptiMUSGPT-5.6 SolIterative CoT1241K290
OptiMUSGPT-5.6 SolEffiOR1666K420
OptiMUSGLM-5.2Iterative CoT1446K325
OptiMUSGLM-5.2EffiOR1979K485
OptiMUSQwen3.7-MaxIterative CoT1136K270
OptiMUSQwen3.7-MaxEffiOR1558K390
LLMOPTGPT-5.6 SolIterative CoT1552K360
LLMOPTGPT-5.6 SolEffiOR2191K550
LLMOPTGLM-5.2Iterative CoT1344K310
LLMOPTGLM-5.2EffiOR1874K460
LLMOPTQwen3.7-MaxIterative CoT1032K255
LLMOPTQwen3.7-MaxEffiOR1454K370
Average—Iterative CoT12.5041.83K301.67
Average—EffiOR17.1770.33K445.83

Over the six settings, Iterative CoT requires an average of 12.50 LLM calls, 41.83K tokens, and 301.67 seconds per task. The corresponding numbers for EffiOR are 17.17 calls, 70.33K tokens, and 445.83 seconds. Thus, EffiOR does incur additional cost during refinement.

However, this cost is incurred before the actual solving process. The LLM analyzes and modifies the solver program, but is not involved when the resulting program is executed. The optimized program can be saved and reused without repeating the refinement process. This allows the initial cost to be amortized across multiple executions. We do not assume that the additional cost is recovered when the program is used for only one solve.

W3. Only two optimization modeling systems were evaluated

Our current experiments use two upstream modeling frameworks, OptiMUS and LLMOPT, together with three refinement backbones. We chose these frameworks to evaluate EffiOR with different upstream modeling workflows. EffiOR operates on the solver program produced by an upstream system, rather than depending on its internal modeling procedure. If another modeling system provides compatible solver code, the same refinement process can be applied. The mathematical formulation and natural-language description can provide additional information to help the LLM understand the problem, but they are not required for this code-based refinement process. Thus, the approach is not specific to OptiMUS or LLMOPT. The systems mentioned by the reviewer, such as ORLM, OptMATH, SIRL, and StepORLM, could also be used as upstream sources if their generated solver programs are compatible with our execution interface. We agree that evaluating more systems would provide further evidence. Our current results cover two different modeling frameworks and three refinement backbones. We will make this scope clear in the revised manuscript.

W4. The evaluation may be flawed

All programs in our experiments were run in the same isolated environment, using Python 3.13 and Gurobi 12, with the same maximum four-core allocation, memory limit, and solving time limit. We also checked the invocation of the Solver Thread and Portfolio Selection Skill. It was used in 3 OptiMUS tasks and 5 LLMOPT tasks, accounting for 1.68% and 2.79% of the 179 tasks, respectively. Explicit changes to solver thread or portfolio settings were therefore relatively infrequent. We agree that the same four-core limit does not guarantee that every solver uses exactly the same number of threads. The invocation statistics also do not establish identical default thread usage. We will state this limitation in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.