- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EffiOR improves an existing operations-research formulation and solver program rather than generating a model from scratch. It aligns the problem description, formulation, and code through a semantic intermediate representation, selects among 29 established optimization skills using 39 shared selection features, and updates the representations after each transformation. Instrumented solver runs support objective and stricter solver-state consistency checks. Experiments combine two upstream modeling frameworks with three refinement backbones on FrontierOR-179, plus consistency evaluations on four smaller modeling benchmarks. EffiOR outperforms a generation-budget-matched Iterative CoT baseline in all six reported FrontierOR settings. It improves average QTE over the original models by 14.24 percentage points, although quality gains vary across settings.
Critical strengths
- The paper identifies a useful downstream task and provides a concrete implementation of established OR knowledge for LLM agents. The skill catalog includes applicability conditions and formulation/code examples, while the evolving IR records how solutions map back to the original model. Shared necessary-condition screening is a sensible way to reduce repeated applicability reasoning. The contribution is the organized application and checking of these transformations, rather than a claim to have invented the underlying OR techniques.
- The evaluation separates upstream modeling from refinement by freezing the input artifacts, and includes a same-backbone, generation-budget-matched baseline. The strongest setting improves OptiMUS QTE from 35.75% to 56.42%, and the skill-category ablation locates most of the gain in formulation refinement. Reporting stricter state consistency alongside objective agreement is valuable: the 94.47% versus 88.73% macro averages on the smaller benchmarks expose discrepancies that objective agreement alone would hide. The manuscript also acknowledges quality regressions for some LLMOPT/backbone combinations.
Critical weaknesses
The efficiency and consistency results are reported separately, leaving the usable benefit of verified transformations unclear. On FrontierOR, strict consistency ranges from 83.80% to 95.53%, so a nontrivial fraction of outputs fail the paper's stronger check. The paper does not clearly report the intersection of QTE improvements with successful consistency checks, or performance after falling back to the original artifact when a check fails. Moreover, comparing objectives within 1% and matching solver status can flag a valid reformulation that finds a better incumbent or proves optimality within the time limit. Conversely, agreement on a finite execution does not establish general semantic equivalence. A failure breakdown is needed to distinguish semantic errors from conservative rejections and valid improvements.
Questions for authors
- Can you report, per setting, QTE and quality after rejecting inconsistent transformations and reverting to the frozen baseline? Please also report how many efficiency gains pass the strict check and break down failures into semantic violations, execution failures, and valid changes in objective or solver status.
- How does the verifier handle an exact reformulation that improves a time-limited incumbent by more than 1%, changes TIME_LIMIT to OPTIMAL, or replaces one solve with a decomposition requiring several solves? Please state which checks compare semantic outcomes and which require identical execution states or sequences.
- Can a matched-budget control use the same 29 skills but check applicability directly, without the shared-feature routing layer? An IR ablation or routing-cost measurements would clarify which part of the proposed mechanism contributes beyond supplying OR knowledge.
- What are the actual generation budgets, number of solver executions, and refinement/verification costs for EffiOR and Iterative CoT? Please distinguish final-program runtime from total one-off refinement cost and discuss when repeated use amortizes that cost.
3 comments
Question 2:
We use small-scale instances to examine whether the transformations preserve the original mathematical problem. These instances are chosen so that both the original and transformed formulations can be solved to proven optimality within the available budget. For each instance, we examine the correspondence maintained by the IR, including the mappings of variables, constraints, and objective terms. We then map the transformed solution back to the original variable space, check its feasibility against the original constraints, and compare the optimal objective values. We do not regard agreement of optimal objective values alone as sufficient. It must be accompanied by a valid IR mapping and feasibility preservation. For transformations involving auxiliary variables or variable elimination, we check the corresponding solution-recovery or projection mapping rather than requiring a one-to-one correspondence between all variables. These small-instance checks provide a direct way to examine mathematical preservation without the interference of time-limited solver outcomes, since both formulations are solved to proven optimality. For larger instances, a better incumbent or a change from TIME_LIMIT to OPTIMAL does not by itself imply an invalid reformulation. Such cases need to be assessed through the recovered solution, original-space feasibility, objective evaluation, and, where optimality is claimed, the corresponding bound or optimality evidence. Our current strict State Consistency criterion also checks solver states and solve sequences. It is therefore a conservative execution-level measure. A transformation that changes the number of solver calls or the solver status may fail this strict check even when its mathematical formulation is valid. We distinguish this execution-level criterion from mathematical equivalence.
Question 3:
We conducted an additional ablation to examine the contribution of the shared feature-screening mechanism. Direct Skill Checking retains the same IR and all 29 Optimization Skills, but removes the 39 shared Selection Features and their associated Skill Gates. Instead, it directly checks the original 117 Skill-specific applicability conditions. The results show a clear difference. Direct Skill Checking achieves 38.55% QTE, while full EffiOR achieves 56.42%, an increase of 17.87 percentage points. Qual@2% and Qual@1% also increase from 60.34% and 54.19% to 75.98% and 69.83%, respectively. Objective and state consistency improve from 93.85% to 97.33% and from 89.94% to 95.53%. The 39 shared Features are not replacements for the original applicability conditions. They capture necessary structural conditions shared by different Skills and are used for preliminary screening. A Skill that passes its gate still undergoes its complete applicability check.These results suggest that the feature-screening and Skill-gating mechanism contributes beyond simply providing the same OR knowledge. We acknowledge that this ablation retains the IR and therefore does not independently measure the contribution of IR alignment.
| Method | Qual@2% ↑ | Qual@1% ↑ | QTE ↑ | Obj. Cons. ↑ | State Cons. ↑ |
|---|---|---|---|---|---|
| EffiOR (Full) | 75.98 | 69.83 | 56.42 | 97.33 | 95.53 |
| Refinement + Solver | 73.18 | 67.60 | 54.75 | 97.33 | 95.53 |
| Solver Only | 61.45 | 54.19 | 35.20 | 93.17 | 89.39 |
| Direct Skill Checking | 60.34 | 54.19 | 38.55 | 93.85 | 89.94 |
| Original | 62.57 | 54.19 | 35.75 | — | — |
Question 4:
We measured the refinement cost of EffiOR by stage, including LLM calls, token consumption, and wall-clock time.The four stages are standardized structural extraction, feature evaluation and Skill invocation, ordered Skill application, and consistency verification. Their total cost is 17.17 LLM calls, 70.33K tokens, and 445.83 seconds per task. For comparison, Iterative CoT requires an average of 12.50 calls, 41.83K tokens, and 301.67 seconds. Thus, EffiOR has a higher one-off refinement cost. The break-even point therefore depends on the actual runtime reduction and the number of subsequent executions.We distinguish LLM calls from solver executions, since they measure different operations. Solver-execution counts and final-program runtime are reported separately from the one-off refinement cost.
| Refinement Stage | LLM Calls | Total Tokens | Wall-clock (s) |
|---|---|---|---|
| 1. Standardized Structural Extraction | 3.434 | 14.77K | 115.916 |
| 2. Refined Feature Evaluation and Skill Invocation | 1.717 | 6.33K | 35.666 |
| 3. Ordered Skill Application | 8.242 | 34.46K | 182.790 |
| 4. Consistency Verification | 3.777 | 14.77K | 111.458 |
| Total Refinement | 17.170 | 70.33K | 445.830 |
We sincerely thank the reviewer for the careful evaluation and constructive feedback.
Weaknesses:
Thank you for pointing out the need to better explain the relationship between solving efficiency and transformation consistency. In our experiments, we use a strict evaluation protocol in which State Consistency is a prerequisite for counting a transformed result as successful in the solution-quality and efficiency metrics. Specifically, for each instance in FrontierOR-179, we first perform strict consistency verification between the original and transformed programs. Only transformations that pass the State Consistency check are counted toward Qual@2%, Qual@1%, and QTE. If a transformation fails this check, the instance is counted as unsuccessful, even if its objective value or runtime appears better. These instances remain in the original denominator of 179 rather than being removed from the evaluation. Therefore, our reported solution-quality and efficiency results have already undergone strict consistency filtering. For example, with OptiMUS and GPT-5.6 Sol, 171 out of 179 instances pass State Consistency, among which 101 also satisfy the solution-quality and runtime requirements of QTE. Thus, the reported QTE of 56.42% includes only results that have passed strict consistency verification. If QTE were calculated only over instances that pass State Consistency, excluding the failed instances from the denominator, the resulting conditional QTE rate would be higher. For the above setting, it would be 101/171 = 59.06%, compared with 56.42% when all 179 instances are included. We retain all 179 instances in the denominator to avoid inflating the overall performance by excluding cases that fail verification. When a consistency check fails during transformation, the agent further examines its internal transformations, locates the inconsistent variables, constraints, or objective expressions, and attempts to repair them before performing verification again. Results that still fail the final strict check are not counted as performance improvements. Thus, our reported QTE and solution-quality metrics reflect solving improvements that pass strict consistency verification over the complete test set, rather than potential gains from unverified transformations.
Question 1:
Thank you for pointing out the need to better explain the relationship between solving efficiency and transformation consistency. In our evaluation, strict State Consistency is a prerequisite for counting a transformed result as successful in the solution-quality and efficiency metrics. Specifically, if a transformed result fails the strict consistency check, it is counted as unsuccessful, regardless of its observed objective value or runtime. Such instances remain in the original denominator of 179 rather than being removed from the evaluation. Therefore, our reported solution-quality and efficiency results have already undergone strict consistency filtering. For example, with OptiMUS and GPT-5.6 Sol, 171 out of 179 instances pass State Consistency, among which 101 also satisfy the solution-quality and runtime requirements of QTE. Thus, the reported QTE of 56.42% includes only results that have passed strict consistency verification. The same rule applies to all six framework–backbone settings. If QTE were calculated only over instances that pass State Consistency, excluding the failed instances from the denominator, the resulting conditional QTE rate would be higher. For the above setting, it would be 101/171 = 59.06%, compared with 56.42% when all 179 instances are included. We retain all 179 instances in the denominator to avoid inflating the overall performance by excluding cases that fail verification. When a consistency check fails during transformation, the agent further examines its internal transformations, locates the inconsistent variables, constraints, or objective expressions, and attempts to repair them before performing verification again. Results that still fail the final strict check are not counted as performance improvements. In this way, an improvement is counted only when it passes both the consistency check and the corresponding quality or runtime requirement.