acceptodds
Under review as a conference paper at ICLR 2027

EvoORSkill: When Optimization Skills Evolve, Who Checks The Math?

Abstract

Large language models can translate operational requirements into optimization models and solver code, but code that runs successfully may still solve the wrong decision problem. Reusing generated procedures reduces repeated modeling work, yet adapting them to new requirements can change which decisions the model allows without causing execution errors. We introduce EvoORSkill, an agent framework that evolves reusable operations research skills together with their revision strategies, without retraining the underlying LLM. Each skill combines an executable modeling procedure with explicit mathematical relationships, applicability conditions, and version information. EvoORSkill organizes these skills in an evolving skill hypergraph (ESH), which connects the procedures assembled for a task through the mathematical objects they share. When the agent proposes a revision in response to task feedback, ESH traces the affected relationships in both the accepted and candidate hypergraphs. Specification, requirement, and retention checks then determine whether the revised skills can enter the reusable library. Historical replay compares revision strategies using records of past attempts with compatible versions, avoiding repeated candidate generation and evaluation. The selected strategy guides further skill revisions, whose outcomes extend the history used to refine the strategy. Across 518 problems from six datasets, EvoORSkill achieves 81.5% average solution accuracy, exceeding the strongest evaluated baseline by 5.0 percentage points. Removing cross-skill dependency checking or specification and requirement checks reduces accuracy by 6.6 and 13.3 percentage points, respectively. Historical replay reduces library-building tokens by 14.8% compared with generating and evaluating candidates afresh. With its evolved library frozen, EvoORSkill achieves 57.0% accuracy on 100 OptMATH problems, outperforming the same agent without a library by 25 percentage points while using 28.3% fewer online tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Natural-language optimization, Large language models, Self-evolving, Reusable agent skills, Verifier-gated execution, Mathematical programming

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1dq6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoORSkill studies how reusable optimization modeling skills can evolve while preserving their intended mathematical meaning. It represents versioned skills as hyperedges over shared mathematical objects, traces dependencies affected by revisions, and applies specification, requirement, and retention checks before admitting updates. Historical replay improves revision strategies by reusing compatible evaluation records. Experiments across six datasets and frozen-library transfer to OptMATH report improvements in solution accuracy and token efficiency.

Critical strengths
  1. The paper identifies an important challenge: revising one modeling component can affect other components through shared mathematical objects, even when the resulting code remains executable.
  2. Connecting skill evolution with dependency-aware mathematical validation is a promising direction for reliable optimization modeling agents.
  3. The component ablations and frozen-library transfer experiment provide useful evidence supporting the evaluated framework.
Critical weaknesses
  1. The central validation procedures are insufficiently specified. Section 3 defines specification, requirement, and retention checks, but does not explain their implementations in sufficient detail. It remains unclear how requirement records are constructed and confirmed, how executable model fragments are compared with declared mathematical relations, and how retained tasks are selected. These procedures are central to the claimed reliability benefits.
  2. The presentation and missing experimental details substantially limit reproducibility. Section 3 moves between task-level models, skill collections, and revision strategies using dense notation without a complete worked example. The supplied manuscript has no implementation appendix resolving these ambiguities. Task splits, initial-library construction, prompts, revision budgets, solver settings, and checker configurations are insufficiently documented. Regenerating aggregate tables does not establish reproducibility of the underlying agent runs.
  3. The computational cost of revision checking is not sufficiently characterized. The paper reports aggregate library-building tokens and ablation totals, but does not separately report the costs of specification, requirement, and retention checks or clarify which operations invoke an LLM. Solver-based checks also require runtime accounting beyond token usage. Differences between ablation totals cannot isolate checking overhead because removing checks changes candidate admission and subsequent evolution. Consequently, the computational trade-off underlying the reliability gains remains unclear.
  4. Cross-problem dependencies and conflicts are insufficiently analyzed. The method primarily explains dependencies among components assembled for a single task. Retention checks and frozen-library transfer provide cross-task evidence, but do not explain how revisions to shared skills propagate across different problem families or how conflicting applicability conditions are handled. This limits the conclusions about the broader evolution of a reusable skill library.
  5. The ablations do not fully isolate the proposed mechanisms. Removing cross-skill checking supports checking related components, but does not establish the specific benefit of a hypergraph over an ordinary dependency graph or full rechecking. Specification and requirement checks are removed together, leaving their individual contributions unresolved. Repeated-run variability is also not reported.
Questions for authors
  1. How are requirement records constructed and confirmed? What roles do humans, LLMs, and reference specifications play, and how are the three admission checks implemented?
  2. Can the authors provide a complete revision example showing the original and revised code and mathematical relations, affected dependencies, detected counterexamples, and admission decision?
  3. Please break down token usage, solver runtime, and invocation counts for the three checks. Which operations use an LLM, and which costs are included in the reported build-token totals?
  4. How are cross-problem effects and conflicting applicability conditions handled when a shared skill is revised? Can the authors provide a concrete example?
  5. Can the authors provide reproducible experimental configurations and clarify the benefit of hypergraph-based tracing relative to an ordinary dependency graph or full rechecking?
Reviewer_sHUJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoORSkill, an LLM-agent framework for optimization modeling that builds and evolves a library of reusable modeling skills without retraining the underlying model. The authors argue that when generated modeling procedures are reused and revised for new tasks, a revision can silently change the meaning of the mathematical model while the code still runs. To address this, each skill pairs executable modeling code with its declared mathematical relationships, applicability conditions, and version, and skills are linked in a hypergraph through the variables and parameters they share. When a skill is revised, the framework uses this structure to decide which task requirements need to be rechecked, and admits the revision only if it passes a specification check, a solver-based requirement check that searches for counterexamples, and a retention check on previously solved tasks. A historical replay mechanism additionally reuses past revision records to improve the revision strategy at lower cost. Experiments on six optimization modeling benchmarks show higher average accuracy than existing methods, ablations of the main components, and transfer of the evolved library to OptMATH.

Critical strengths
  • Representing each skill as executable modeling code paired with its declared mathematical relationships is a relatively new design in this line of work, compared with prior libraries that store mainly textual insights or workflow guidance.
  • Checking revisions through solver-based counterexample search is a principled idea, as a pass provides concrete evidence that the candidate model admits no feasible solution violating the checked requirement.
  • The evaluation covers six benchmarks, includes ablations of the main components, and tests transfer of the evolved library to a new dataset.
  • The paper is generally well organized, and the figures help illustrate the overall framework and the role of shared mathematical objects.
Critical weaknesses

Overall, I think this is a solid paper with an interesting idea, and the points below are mainly suggestions that could further strengthen it. I should also note that I am not fully confident in my understanding of some parts, so please take them as questions rather than firm criticisms.

  1. If I understand correctly, the hypergraph is mainly used to decide which requirement records need to be rechecked after a revision, by following the variables and parameters shared between skills. It would be helpful to compare this with a simpler alternative, namely rechecking all requirements of the task after every revision. The current ablation nicely shows that checking beyond the edited skill improves accuracy, but it would be even more convincing to also show what the hypergraph adds over a full recheck, for example in terms of the number of checks saved or the overall cost. In addition, a motivating example in which a revision to one skill actually makes another skill incorrect might illustrate the role of the hypergraph more clearly than the current facility-location example. Could the authors comment on this?
  2. The paper is motivated by the risk that a revised model may still run and even produce the correct answer while being mathematically wrong. Since the main evaluation is based on solution accuracy, which compares only the final objective value, some additional evidence that the checks indeed catch such hidden errors would make the contribution clearer. For instance, it would be nice to see how many revisions were rejected by the specification or requirement checks despite executing normally, a few concrete examples of such cases, or how often each check returns FAIL or UNKNOWN. Re-solving some test problems with different data could also be a simple way to show that the resulting models remain correct. Would the authors be able to share some evidence of this kind?
  3. It would help to describe more clearly how the problems used for evolving the library relate to the test problems. Section 4.4 describes the library as evolved on the six source datasets, while the test set includes 100 problems from IndustryOR and 18 from ComplexOR, which, if I am not mistaken, correspond to the full size of these two datasets. A short description of the evolution, development, and test splits for each dataset would make the results easier to interpret, especially for IndustryOR, where EvoORSkill shows its largest gain. Could the authors clarify this?
Questions for authors

In the ablation, historical replay reduces build tokens, but it also improves accuracy by 4.4 points compared with fresh evaluation. Since replay is described as reusing recorded outcomes rather than changing the revision process itself, could the authors explain where this accuracy gain comes from?

Reviewer_mYpfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper highlights an important but underexplored aspect of OR skill evolution: mathematical dependencies may exist across different skills, making isolated skill revisions potentially unreliable. By integrating skill evolution with requirement-aware and dependency-aware mathematical validation, the paper offers a promising new perspective on improving the reliability and capabilities of optimization modeling agents.

Critical strengths

The idea of evolving reusable modeling skills rather than relying solely on one-shot model generation is promising. Incorporating dependency-aware validation into this process is an interesting direction toward improving the reliability of optimization modeling agents.

Critical weaknesses
  1. The motivation for ESH is insufficiently established. The “exactly once → at most once” example illustrates a local formulation error and shows that executable code can violate task requirements. However, it does not demonstrate why checking the edited skill is insufficient or what additional errors dependency-based checking would detect. A concrete example would help justify this design.

  2. The framework relies on Requirement Records to assess whether LLM-generated models satisfy business requirements and to guide skill revision, yet the paper fails to explain how these records are generated in practice. This checking mechanism is inherently limited to recorded requirements, leaving errors involving unrecorded requirements potentially undetected. The paper does not establish that these records adequately capture the actual business requirements.

  3. The benefit of ESH is not clearly demonstrated. As described, ESH traces dependencies to select requirements for counterexample checking. Why not directly validate all requirements against the revised model? If dependency-based selection is intended to reduce verification costs, the paper should compare it against exhaustive validation, accounting for ESH construction and maintenance costs, verification costs, and accuracy.

  4. Important technical and experimental details are missing. The paper does not explain how the initial skill library is constructed or how the specification check compares generated model fragments with their mathematical declarations. Essential experimental settings, including solver configurations, are also unspecified.

  5. What exactly is included in online tokens? Removing skill evolution increases reported online tokens, but evolution itself involves numerous stages: ESH construction, specification/requirement/retention checks, candidate skill generation, revision strategy generation, and historical replay. The paper does not clearly identify the LLM calls and token costs associated with these stages or compare the total evolution costs with those of the baselines. Consequently, it remains unclear whether the accuracy gains depend on substantially greater token expenditure.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.