acceptodds
Under review as a conference paper at ICLR 2027

OR-Skill: Self Evolutionary Agent Skills for Optimization Modeling

Abstract

Optimization modeling is crucial for decision-making. Constructing optimization models from natural language descriptions has long been a challenging research goal. Prior work has predominantly relied on fine-tuned large language models (LLMs) or LLM-based agentic workflows to automate this process. However, the rise of general-purpose coding agents, such as Claude Code, offers a new paradigm for tackling such complex tasks. In this paper, we present OR-Skill, a framework that equips a general-purpose agent with a specialized skill for optimization modeling, where the skill is evolved iteratively in an environment built from our curated benchmark. The evolution process operates in an iterative loop. In each iteration, the agent executes tasks with a candidate skill, receives feedback, attributes failures to their root causes, and summarizes them into consolidated diagnostic reports at different error levels. Based on these reports, we perform cascade evolution in reverse order of error severity to refine the skill. We conduct extensive experiments across seven optimization modeling benchmarks of varying difficulty, on which OR-Skill achieves state-of-the-art or comparable performance on all benchmarks. The improvements are particularly significant on challenging benchmarks: IndustryOR (+ absolute accuracy) and ComplexLP (+).

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Optimization Modeling, LLM Agents, Agent Skills, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3XQsOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces OR-Skill, an approach that equips a general-purpose coding agent (Claude Code running DeepSeek V3.2) with a specialized optimization-modeling skill set consisting of a SKILL.md file, reference files, and solver scripts. The skill is refined offline via an evolutionary loop using "Hard100"—a newly curated dataset of 100 complex optimization problems with expert ground-truth models.

The evolutionary process operates in rounds:

  1. A parent skill is sampled from a pool based on its current score.
  2. The agent runs the parent skill on Hard100.
  3. An LLM judge categorizes failures into four distinct levels (Correct, Syntax, Improvable, Logical).
  4. Failure modes across levels are aggregated into a root-cause report.
  5. An LLM updates the skill via Git commits in reverse severity order (L3→L2→L1L_3 \to L_2 \to L_1, termed "cascade evolution"), generating N=5N=5 child skills.

Child skills are evaluated using accuracy minus a token-length penalty (Eq. 3). After T=12T=12 rounds, the top-performing skill is selected and frozen.

The authors claim three main contributions:

  • The skill-evolution pipeline (level-based grading, root-cause attribution, cascade editing, and length-regularized selection).
  • The Hard100 benchmark.
  • The open-sourced final skill.

The method is evaluated across seven subsets from six standard benchmarks (NL4Opt, NLP4LP, ReSocratic, MAMO EasyLP/ComplexLP, ComplexOR, IndustryOR), using GPT-5 to judge correctness. The authors report state-of-the-art or tied performance across all subsets—for instance, achieving 91.0% vs. 82.1% (EvoSkill) on ComplexLP and 78.6% vs. 73.8% on IndustryOR. The submission also includes ablations, a judge-human agreement study, an embedding leakage check, and case studies.

Critical strengths
  1. The component ablation in Table 8 is striking: the vanilla agent achieves 56.8% on ComplexLP, whereas the fully evolved skill boosts performance to 91.0%. This shows that a well-crafted domain skill does heavy lifting on hard instances. If the protocol issues raised below can be resolved, this is a very practical takeaway—it suggests that pairing a frozen agent with an inspectable, learned skill can rival or outperform specialized, fine-tuned OR models.

  2. Because the skill is managed as a Git repository, every iteration produces a readable diff. I appreciate that the paper openly displays:

  • The structure of the final skill (Fig. 5).
  • The step-by-step update history (Table 6).
  • Detailed token counts (Table 7).
  • Qualitative case studies, including an explicit breakdown of a failure case (D.5).

This transparency makes it easy to see what the agent actually learned and where it still struggles, which is vital for real-world OR deployments.

Critical weaknesses

1. Substantial unacknowledged overlap with prior work weakens the novelty claims. The core methodology leans heavily on concepts already introduced in prior literature—most notably AlphaOPT (Kong, Qu et al., arXiv:2510.18428, Oct 2025), which is missing from the references. Specifically, AlphaOPT already:

  • Learns reusable optimization knowledge offline using solver feedback from failed attempts.
  • Formulates library building as max⁡ℓEt[Success(t∣ℓ)]−λ Ω(ℓ)\max_\ell \mathbb{E}_t[\mathrm{Success}(t\mid\ell)] - \lambda\,\Omega(\ell), which is functionally identical to the token-regularized objective in Eq. 3.
  • Uses progressive disclosure mechanisms for knowledge retrieval.
  • Groups failure symptoms by root cause into single diagnosed issues (directly analogous to Sec. 3.6).
  • Builds a taxonomy that closely mirrors Table 5 (flow conservation, inventory balance, big-M formulation, continuous vs. discrete handling, index mismatches).
  • Evaluates on the exact same benchmarks (NL4OPT, NLP4LP, IndustryOR, MAMO ComplexLP).

AlphaOPT even explicitly suggests extending its library approach into agent skills in its conclusion. Beyond AlphaOPT, several other relevant works are unmentioned:

  • OptiMind (arXiv:2509.22979): Error analysis converted into reusable optimization hints.
  • ReLoop (arXiv:2602.15983): Verification of silent formulation errors (L3L_3 in this paper).
  • NEMO (arXiv:2601.21372) and OptimAI / OR-LLM-Agent: Agentic frameworks for OR modeling.
  • StepORLM (ICLR 2026): Self-evolving OR modeling.
  • General skill-evolution frameworks like SkillOpt and Trace2Skill.

Once these works are accounted for, the remaining novelty is largely constrained to packaging knowledge specifically into a coding-agent skill, population-based selection, and the specific reverse-cascade editing order.

2. Mathematical inconsistencies in the selection objective and flaws in evaluation protocol.

(a) Equation 3 breaks under the reported numbers. Plugging in the given weights (w1=10−3w_1=10^{-3}, w2=2×10−4w_2=2\times10^{-4}) and the token lengths of the final skill (3,817 and 7,427 from Table 7), the length penalty evaluates to:

3.82+1.49≈5.303.82 + 1.49 \approx 5.30

Since accuracy is bounded in [0,1][0, 1], the final skill receives a total score around −4.5-4.5. Meanwhile, the baseline s0s_0 would score drastically higher due to its minimal length. Under a strict arg⁡max⁡\arg\max, Eq. 3 would always revert to s0s_0. Furthermore, sampling proportional to scores becomes ill-defined when scores are negative. As presented, the selection math couldn't have generated the reported results, making it difficult to trust the Occam ablation.

(b) Potential evaluation artifacts inflating margins.

  • Permissive error tolerances: The evaluation uses a 1% relative tolerance, whereas standard literature generally uses 10−610^{-6}.
  • LLM Judge Bias: The GPT-5 judge appears to favor the proposed method. Switching to exact answer extraction causes OR-Skill's score to drop by 9.0 points, compared to only 6.3 for OptiTree (Table 9).
  • Inconsistent denominators in Table 1:
    • On IndustryOR, agent baselines use k/42k/42, while training baselines use k/46k/46.
    • On ComplexOR, 71.4% corresponds to 10/1410/14, but EvoSkill's reported 69.1% doesn't map to any k/14k/14, and the reported +2.3%+2.3\% gain is smaller than a single problem instance.
  • Confounded baselines & missing controls: Fine-tuning baselines rely on a smaller backbone (Qwen3-8B), and the direct control—Claude Code + DeepSeek V3.2 without any skill—is missing from Table 1.
  • Lack of variance reporting: Results are reported without seeds or standard deviations. The cascade ordering claim is evaluated against a single pass rather than a forward L1→L3L_1 \to L_3 baseline, and on EasyLP, two ablated variants perform no better than the bare agent.
Questions for authors
  1. What were the exact Eq. 3 scores for s0s_0 and s∗s^*? Was the final skill picked purely by Hard100 accuracy or by Eq. 3? How did you handle score-proportional sampling when raw scores went negative?
  2. Could you explicitly map out which components are novel relative to AlphaOPT (particularly Eq. 3 vs. its size-regularized objective, and Sec. 3.6 vs. its root-cause diagnosis)? Are you able to provide a direct comparison against AlphaOPT using a matched backbone?
  3. Can you benchmark all methods on identical instance sets using a standard 10−610^{-6} tolerance and deterministic answer extraction, reporting mean ±\pm std across at least 3 seeds? Could you also clarify the denominator discrepancies on IndustryOR (k/42k/42 vs k/46k/46) and ComplexOR in Table 1?
  4. Can you include (a) a forward-cascade ablation (L1→L3L_1 \to L_3) and (b) a base row for Claude Code + DeepSeek V3.2 without any skill in Table 1? How were EvoSkill and OptSkills set up in terms of data, execution budgets, and runtime environment?
  5. The caption in Fig. 6 notes that Hard100 counts are "pending verification," yet Sec. C.2 draws quantitative conclusions from them. Are these counts verified and finalized?
  6. In Table 9, how was a "human expert consensus" accuracy established for the full ComplexLP dataset if human annotation was only performed on a 50-instance subset?
1 comment
AuthorsOpenReview ↗

Thank you for your detailed review!

Thank you for your detailed review. We address your concerns below.

W1/Q2. Novelty relative to AlphaOPT. We apologize for omitting AlphaOPT and will cite it in the revision.

We acknowledge that Eq. 3 shares AlphaOPT's performance–size regularization principle, and Sec. 3.6 shares its root-cause consolidation principle. We do not claim either as novel in isolation.

OR-Skill integrates these principles into population-based evolution of workflow instructions, domain references, and executable utilities. Its specific update mechanism combines expert-model-based L₀–L₃ grading, level-wise diagnosis, and an L₃→L₂→L₁ editing cascade.

W1.3 Comparison with AlphaOPT. We have also added an experiment comparing OR-Skill with our re-implementation of AlphaOPT:

MethodNL4OptIndORCompLP
AlphaOPT, learned on Hard10096.2 ± 1.071.4 ± 4.879.3 ± 0.9
Claude Code + AlphaOPT library96.7 ± 0.872.2 ± 3.681.1 ± 1.8
OR-Skill98.0 ± 0.478.6 ± 4.891.0 ± 1.8

W1.4 Originality. Our contribution is the integrated framework for severity-structured evolution of Git-versioned agent skills and its empirical benefits, rather than the independent novelty of each component. We will revise our claims accordingly.

W2.1 Eq. 3 (W2(a), Q1). See G1.1 and G1.2. s* was selected by Eq. 3, and the scores of s₀ and s* are given in our response to sHSS (Q1.2).

W2.2 Error tolerance. We apologize for the mistake: we use an absolute tolerance of 0.01, not a relative tolerance. We will correct this in the revision. We also report additional rescoring at a relative tolerance of 10⁻⁶ (G4.2, G4.3).

W2.3 LLM judge bias. We understand this concern. However, the expert audit and cross-scorer results do not indicate systematic bias in favor of OR-Skill.

ComplexLP (111)ExpertsGPT-5 judgeExtraction
OR-Skill102101 (−1)91 (−11)
OptiTree8283 (+1)76 (−6)

GPT-5 undercounts OR-Skill and overcounts OptiTree by one instance relative to expert consensus. All five label disagreements per method are reported in sHSS Q2.5. The extractor (κ = 0.61) fails more often on OR-Skill because of output-parsing errors. Our programs follow the skill's verification step and print solver status, variable values and verification results, so a rule-based extractor may select the wrong numerical value. App. D.3, for example, prints the optimum (1,000) and then the units transported (1,840). Canonical-format numerical comparison preserves the four-method ordering (G4.3).

W2.4 Denominators, variance, and controls. G2 addresses the denominator discrepancies, variance reporting and the missing no-skill control. The backbone confound is addressed in our response to eAto (W3.1–W3.2).

W2.5 Cascade order. Results for the cascade order (mean ± sample standard deviation over three evaluation seeds):

OrderEasyLPCompLPIndOR
L₃→L₂→L₁ (ours)98.1 ± 0.591.0 ± 1.878.6 ± 4.8
L₁→L₂→L₃94.3 ± 0.888.3 ± 2.776.2 ± 4.1
Random order95.8 ± 1.089.2 ± 1.876.2 ± 7.1

For the random-order variant, one of the six permutations is sampled uniformly and independently for each child's cascade, using a fixed offline random-number generator. The resulting skill is evaluated with three inference seeds.

W2.6 EasyLP ablations. On EasyLP, the original single-run result for the bare agent in Table 8 is 92.1, and the new three-seed mean is 92.2. Without the cascade or the Occam term, most of the EasyLP gain disappears while much of the ComplexLP gain remains, consistent with their role in reducing interference on easy tasks.

Q4. Setup of EvoSkill and OptSkills. Q4(a) and (b) are answered in W2.5 and G2.2. EvoSkill and OptSkills are re-run from their official implementations with DeepSeek-V3.2 and Hard100, using the same sandbox, tools, solvers, judge and held-out instance sets. The shared learning cap is 7,200 task executions and up to 7,200 associated feedback judgments per method, excluding initialization and retries, with test-set feedback forbidden. For OR-Skill this is 12 rounds × (1 parent + 5 children) × 100 tasks; its additional 36 attribution and 180 editing calls are reported separately. Matching task executions does not equalize all LLM calls, tokens or wall-clock cost.

Q5. Fig. 6 counts. See our response to eAto (W2(iii)).

Q6. Human consensus on full ComplexLP. See G4.1.

Ethics. We will credit AlphaOPT for the overlapping principles and clarify our novelty claims. Fig. 6 is recomputed from the Hard100 judgment records, and Sec. C.2 is updated accordingly.

We hope this addresses your concerns.

Reviewer_eAtoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies natural-language-to-optimization-model translation with a general-purpose coding agent (Claude Code driven by DeepSeek-V3.2) that is equipped with a domain skill — a git-versioned package of SKILL.md, reference notes and solver scripts — and asks how that skill can be improved automatically. OR-Skill runs the agent on Hard100, a new set of 100 hand-written, expert-annotated problems; a GPT-5 judge grades each output as correct, syntax error, improvable or logical error (L0–L3); failures at each level are clustered into a root-cause report; and DeepSeek-R1 edits the skill level by level in reverse severity order (L3 → L2 → L1) as git commits, with children selected from a skill database by an accuracy-minus-length score (Eq. 3, Alg. 1). Starting from a minimal meta-prompt, 12 rounds yield an 11.8k-token skill that is best or tied-best on seven subsets of six benchmarks, e.g. 78.6% on IndustryOR, 91.0% on ComplexLP and 71.4% on ComplexOR versus 73.8%, 82.1% and 69.1% for EvoSkill (Table 1), supported by ablations, an out-of-distribution test and judge and leakage checks in the appendix.

Critical strengths

S1. The comparison is broad and the gains on the hardest subsets are substantial: 14 baselines spanning direct prompting, five agentic workflows, two skill-learning agents and six training-based models (Table 1), with OR-Skill best on all seven subsets and 8.9 points ahead of the strongest baseline on ComplexLP. All agent-based methods share one base model (Sec. 4.1), which keeps those comparisons controlled.

S2. Each design component has a matching ablation (Table 2: removing root-cause attribution drops ComplexLP from 91.0 to 79.2; removing the cascade drops EasyLP from 98.2 to 91.4; removing the Occam term lowers both), and the paper adds useful robustness evidence: an NL4Opt variant with all numbers removed, where OR-Skill falls from 98.0 to 82.0 while DeepOR falls from 97.7 to 32.7 (Fig. 4); an expert-written skill that reaches 76% and is overtaken at iteration 3 (Sec. 4.5); judge agreement with experts (κ = 0.92, App. C.3); and an embedding-similarity leakage check between Hard100 and every test set (App. C.4).

S3. The update mechanics are engineered for text artifacts rather than borrowed wholesale: failures are clustered into rules that must generalize and may not hard-code answers (Sec. 3.6, App. A.4), and the counter-intuitive reverse-severity cascade is motivated by the observation that easy-to-hard ordering lets the final commits overfit difficult cases (Sec. 3.7) and is supported by the ablation.

Critical weaknesses

W1. The evaluation leaves room for answer leakage and judge leniency: the answer directory is only renamed, which "alone does not enforce a filesystem access boundary" (App. A.1), while the agent has shell and file tools, and no trace audit is reported. Correctness is decided by GPT-5 — the same judge that drives evolution — validated on only 50 instances (Table 9). Suggestion: move references outside the sandbox or audit all traces, and add exact-match accuracy under a canonical output format.

W2. Several numbers do not reconcile, and Table 1 has no variance: (i) IndustryOR agent-based results are multiples of 1/42 but the re-run training-based ones of 1/46, implying different subsets; (ii) with the stated w1, w2 (Sec. 4.1) and token counts (Table 7), Eq. 3's length penalty is ≈ 5.3 against Acc ≤ 1, so the score would favour near-empty skills; (iii) Fig. 6 is still captioned "counts pending verification" (p. 21); (iv) the IndustryOR and ComplexOR margins amount to two problems and at most one. Suggestion: score all methods on one subset, fix Eq. 3's scaling, verify Fig. 6, and report mean ± std over ≥ 3 seeds.

W3. Cross-category comparisons conflate method with model scale (agent methods on DeepSeek-V3.2, re-run training methods on Qwen3-8B; Sec. 4.1), so the training-data-coverage explanation (Sec. 4.2) is untested, and cost is not compared despite 600 agent runs plus judge calls per round over 12 rounds (Table 4). Suggestion: match base models in at least one comparison and report tokens and wall-clock per method.

Questions for authors

Q1. Can you report an audit of all evaluation traces for reads of the renamed answer directory, or rerun the evaluation with references stored outside the sandbox, and give the resulting accuracies?

Q2. Why do the IndustryOR accuracies of the re-run training-based methods correspond to 46 problems while the agent-based results correspond to 42, and does the same mismatch affect ComplexOR? Please re-evaluate all methods on a single subset with seeds and standard deviations.

Q3. What scaling of Acc and the token counts is actually used in Eq. 3, and how sensitive is the selected skill to w1 and w2? Does the final skill change if the length penalty is applied only at selection time?

1 comment
AuthorsOpenReview ↗

Thank you for your insightful and constructive review!

Thank you for your insightful and constructive review. We address your suggestions below.

Q1.1 Answer leakage. We take answer leakage seriously. G3.1 describes our safeguards, and G3.2 reports the re-evaluation and trace-audit results.

Q1.2 Judge leniency. The evolution and evaluation judges use disjoint datasets and different rubrics. On full ComplexLP, GPT-5's correct count differs from expert consensus by one, with five label disagreements per method. The four-method ranking is preserved with a judge from a different model family and with the canonical numerical scorer (G4.1, G4.3).

W2(i)/Q2 Identical instance sets. This was a bookkeeping error. All methods have now been evaluated on the same 42 IndustryOR and 28 ComplexOR instances (G2.1, G2.2).

W2(ii) Eq. 3 scaling. See G1.1.

W2(iii) Fig. 6. The caption of Fig. 6 was inadvertently retained from an earlier draft, and we apologize for the oversight. The revised figure reports raw counts out of 100, recomputed from the Hard100 judgment records. The counts sum to 100 at every stage, correcting the earlier total of 101%.

StageL₀L₁L₂L₃
s₀13372030
Iter. 35071726
s*800713

W2(iv) Small margins. We agree. On IndustryOR and ComplexOR, the margins amount to two problems and at most one, respectively, so we do not claim statistical significance on these subsets. Expert-verified hard instances are scarce in this field.

W3.1 Base model. We use Qwen3-8B for the training-based methods to align with prior work: ORLM, SIRL, DeepOR and OptMATH all train at the 7–8B scale. We considered training on DeepSeek-V3.2, but training a 671B model exceeds our available compute budget. Agent methods require no parameter training, so we use a strong API-accessible backbone.

W3.2 Model scale alone does not explain the gap. In the original paper's single-run results on ComplexLP, DeepSeek-V3.2 with standard prompting (41.4) and as a bare agent (56.8) both trail the 8B DeepOR (67.1); only with the evolved skill does it reach 91.0. Matched-backbone check:

Qwen3-8BEasyLPCompLPIndOR
DeepOR93.1 ± 0.667.6 ± 1.852.4 ± 4.8
Claude Code w/o skill86.2 ± 1.045.6 ± 2.342.9 ± 4.8
Claude Code + OR-Skill93.4 ± 0.677.2 ± 3.264.3 ± 2.4

W3.3 Coverage. We agree that this explanation was not tested in the paper, and we now examine it directly. We rank instances within each benchmark by their maximum similarity to the union of the baselines' public training corpora, then partition them into coverage groups. This reduces confounding by benchmark difficulty and yields groups of 518/517/517 instances. The table reports accuracy. The accuracy of training-based methods decreases as coverage decreases, whereas OR-Skill remains stable.

CoverageDeepORSIRLOR-Skill
High (n=518)90.387.694.4
Mid (n=517)85.383.093.8
Low (n=517)77.076.094.4

Together with the numeric-abstraction test in Fig. 4, these results support the coverage explanation.

W3.4 Inference cost. We report token usage, tool calls and latency below:

Pilot methodTotal tokensOutput tokensTool callsMedian time
Standard4.7k4.3k013.4 s
Agent w/o skill37.8k3.2k3.127.3 s
OR-Skill48.3k3.4k4.133.5 s

Relative to the bare agent, OR-Skill uses 27.8% more total tokens and 6.25% more output tokens, with an additional 6.2 s of median latency. We will include these results in the revision.

For the offline evolution stage, the scheduled OR-Skill learning budget is 7,200 task executions and 7,200 judgments, plus 36 attribution and 180 editing calls, excluding initialization and retries. Because the skill is reused, this offline stage is not repeated for each evaluation query.

Q3.1 Scaling. See G1.1.

Q3.2 Sensitivity to w₁ and w₂. We scale both weights by a factor c; c = 0 is the "w/o Occam" ablation.

c00.5125
N_meta / N_ref7,800 / 15,9005,100 / 10,8003,817 / 7,4272,700 / 6,2001,450 / 3,100
EasyLP92.397.698.297.896.0
ComplexLP87.490.191.089.284.7

For each c, one skill is learned and evaluated once, so this experiment does not estimate variability across evolution runs. Under the unified protocol, c=0 gives 503/545=92.3 and 97/111=87.4, replacing the paper's inconsistent 92.2/87.3 values.

Q3.3 Penalty only at selection time. Applying the penalty only at final selection, with parents sampled using pᵢ ∝ Accᵢ + 0.01, gives 93.6 on EasyLP and 88.3 on ComplexLP, with 20,300 text tokens (6,900 meta + 13,400 reference tokens, excluding scripts). This supports an effect on the search trajectory, although it does not isolate sampling as the sole causal mechanism. Final selection can also reverse a nonzero accuracy difference, as the longest-child example in sHSS Q1.2 shows.

We hope these clarifications and additional results address your concerns.

Reviewer_sHSSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OR-Skill equips a general-purpose coding agent with an optimization-modeling skill containing workflow instructions, domain references, and executable utilities. The skill evolves on Hard100 through error grading, root-cause attribution, cascade updates, and length-regularized selection. Evaluation on seven subsets from six benchmark suites reports improvements over existing modeling agents and skill-learning methods, with additional component ablations and judge-robustness checks.

Critical strengths

The framework addresses a practical problem through reusable modeling knowledge and executable checks. Versioned skill edits and consolidated failure reports make the adaptation process inspectable. The component ablation provides meaningful evidence of the skill's value: ComplexLP accuracy increases from 56.8% for the bare agent to 91.0% with the full skill (Table 8).

The evaluation includes directly relevant skill-learning baselines using the same base LLM for agent methods. OR-Skill reports 78.6% versus 73.8% for EvoSkill on IndustryOR, and 91.0% versus 82.1% on ComplexLP (Table 1). Removing root-cause attribution reduces ComplexLP accuracy to 79.2%, supporting the usefulness of consolidated diagnostic reports (Table 2).

Critical weaknesses

The central selection mechanism is inconsistent as specified. Equation 3, the weights in Appendix A.3, and the token counts in Table 7 imply a length penalty of 5.3024 for the final skill. Since accuracy is explicitly defined in [0,1], its score is at most -4.3024. Algorithm 1 nevertheless samples parents proportionally to score without specifying a transformation into valid sampling weights that favors higher scores. This may reflect a reporting or implementation mismatch, but the actual scoring and sampling procedure must be clarified for reproducibility.

The evaluation does not clearly establish formulation correctness beyond answer agreement. Section 4.1 describes comparing printed outputs with reference answers. Appendix D.1 illustrates the limitation: an initially incorrect gross-flow formulation already returns the reference value of 72 despite producing net flow 54, before subsequent repair. The human audit is helpful, but Appendix C.3 describes 50 audited instances across all seven subsets, whereas Table 9 reports human-consensus accuracy for the full ComplexLP benchmark. The annotation scope and whether experts checked model structure and solution feasibility remain unclear.

The reported skill gains are promising, but I lean toward weak rejection because the selection procedure and the interpretation of modeling accuracy need clarification. Implementation records and a clearly scoped structural audit would most directly affect my assessment.

Questions for authors
  1. What accuracy scale, length penalties, and score-to-probability transformation were used in the actual implementation? Can you reconcile these with Appendix A.3 and Table 7, and provide representative candidate scores and sampling probabilities?

  2. Did the human audit assess formulation structure and solution feasibility, or primarily answer agreement? How were the full-ComplexLP human-consensus accuracies obtained given the described 50-instance audit across seven subsets, and how often did numerically matching outputs contain substantive modeling errors?

1 comment
AuthorsOpenReview ↗

Thank you for your constructive review!

Thank you for your constructive review. We address your questions below.

Q1.1 Scale and score-to-probability transformation. Thank you for pointing this out. G1.1 and G1.2 give the corrected scale, sampling rule and selection criterion.

Q1.2 Representative scores and sampling probabilities. Representative candidates from the 61-candidate skill database (the initial skill plus 12×5 children):

CandidateHard100 AccN_meta / N_refPenaltyScorep (end-of-round-12 snapshot)
s₀13/100124 / 00.0012400.1287600.004024
best, iter. 350/1001,650 / 2,5000.0215000.4785000.014168
longest child82/1009,200 / 18,0000.1280000.6920000.020360
s*80/1003,817 / 7,4270.0530240.7469760.021954

All four probabilities use the same full-population normalizer, Σᵢ[max(Scoreᵢ,0)+0.01] = 34.480066, and the other 57 candidates account for the remaining probability. This is the end-of-round-12 diagnostic distribution.

The extra length of s* over s₀ costs 5.1784 accuracy points, while s* is 67 points more accurate, and the final skill has the highest score, 0.746976. The longest child is two points more accurate on Hard100, but it incurs 7.4976 additional points of length penalty and therefore scores lower. This is a soft accuracy-length trade-off, not merely a tie-breaker.

Q2.1 Limitations of answer-level evaluation. Thank you for raising this point. We acknowledge that evaluating final answers rather than formulations is a limitation of our evaluation and a broader challenge in optimization modeling. We follow the practice used in NL4Opt, MAMO, IndustryOR and OptiBench, which score the final answer. A key challenge is determining whether two formulations are equivalent, particularly when public benchmarks provide final answers but no reference models. The trace in App. D.1 illustrates this potential false positive. In that case, the agent itself detected the 72-vs-54 contradiction and repaired the model before submission; only the final model.py is scored. We will add this discussion to the Limitations section.

Q2.2 Where formulation structure is checked. Beyond agreement on the answer, structure is checked at two stages. During evolution, the judge compares each output with Hard100's expert reference models, and the L₂ level exists specifically to catch outputs that are numerically correct but loosely formulated. During evaluation, the judge reads the question, reference answer and execution output, so structural validation comes from a separate human audit rather than from numerical judging alone.

Q2.3 Scope of the human audit. The audit has two components: a subset of 50 instances stratified across all seven subsets, used to compute κ, and the full ComplexLP set, i.e., all 111 outputs of OR-Skill and all 111 of OptiTree (222 audited programs). The latter gives the accuracies in Table 9: 102/111 = 91.9% for OR-Skill and 82/111 = 73.9% for OptiTree.

Q2.4 The audit checks structure and feasibility. Two OR experts independently checked every program and its execution log against a fixed checklist:

  1. decision variables and their domains;
  2. each constraint (presence, direction, indexing, boundary conditions);
  3. objective sense and coefficients;
  4. solver status and feasibility of the reported solution, re-verified by substituting it into the constraints;
  5. agreement of the optimum with the reference.

An output is labeled correct only if the formulation is valid and the optimum matches. For the 222-program audit, κ between the two experts before adjudication is 0.84 (212/222 raw agreement); this is separate from the paper's judge-versus-human κ on the 50-instance subset.

Q2.5 Modeling errors among numerically matching outputs. Among numerically matching outputs, substantive modeling errors occur in 2/104 OR-Skill and 3/85 OptiTree programs, leaving 102 and 82 valid programs, respectively.

Relative to expert consensus, GPT-5 has 2 false positives and 3 false negatives for OR-Skill, and 3 false positives and 2 false negatives for OptiTree. Each method thus has five label disagreements, even though its net correct-count discrepancy is one. We disclose these errors rather than treating numerical judging as perfect formulation verification. The per-instance contingency tables and audit sheet will be released with the revision.

Q2.6 Toward formulation-level verification. We agree that formulation-level verification is an important next step, and OR-Skill could support it. Because Hard100 includes reference models, it enables differential testing: perturb the instance data and require the generated model and the reference model to agree. Because skills can contain executable scripts, such a checker, or ReLoop-style behavioral checks, could be integrated as tools the agent calls and could also provide structural feedback during evolution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

We thank all reviewers for constructive feedback!

We thank all reviewers for constructive feedback. We address concerns shared across the reviews below.

G1.1 Eq. 3 contains a reporting error; the implementation is consistent. The paper pairs Acc ∈ [0,1] with weights calibrated for Acc in percentage points, so the reported weights are off by a factor of 100. The code uses Acc ∈ [0,1] with w₁ = 10⁻⁵ and w₂ = 2×10⁻⁶ per token, so the final skill's penalty is 0.053024 (5.3024 points), not 5.3024 on the [0,1] scale.

G1.2 Sampling and selection are well defined. Parents are sampled with p_i ∝ max(Score_i, 0) + 0.01, yielding a valid distribution after normalization and a nonzero probability for every candidate. The final skill is selected by argmax Score. The Occam ablation design (w₁ = w₂ = 0) is unchanged; updated results under the unified protocol are in eAto Q3.2. We will correct Sec. 3.8, Alg. 1 and App. A.3.

G2.1 The k/46 denominators were a bookkeeping error. The re-run training-based methods were scored against an outdated 46-instance file. We will correct this in our revision.

G2.2 Methods have been re-evaluated on identical sets with three seeds. Methods in our experiments, including EvoSkill and OptSkills, have been re-evaluated on identical sets under a matched protocol, with three seeds per fixed learned artifact.

Revised Table 1:

MethodNL4OptNLP4LPReSoc.EasyLPCompORIndORCompLP
DeepOR (Qwen3-8B)97.6 ± 0.883.0 ± 0.373.7 ± 0.793.1 ± 0.664.3 ± 0.052.4 ± 4.867.6 ± 1.8
OptiTree97.8 ± 0.284.3 ± 1.176.8 ± 0.694.6 ± 0.564.3 ± 7.150.0 ± 2.474.8 ± 0.9
EvoSkill95.1 ± 0.893.1 ± 0.982.5 ± 0.897.1 ± 0.669.0 ± 2.173.8 ± 2.482.0 ± 2.7
OptSkills74.3 ± 1.292.1 ± 1.770.4 ± 1.197.6 ± 0.265.5 ± 5.571.4 ± 7.152.9 ± 2.3
AlphaOPT (new)96.2 ± 1.092.1 ± 0.684.5 ± 0.997.1 ± 0.464.3 ± 3.671.4 ± 4.879.3 ± 0.9
Claude Code, no skill (new)91.7 ± 1.088.6 ± 1.482.3 ± 1.192.2 ± 0.860.7 ± 7.159.5 ± 0.056.8 ± 3.6
OR-Skill98.0 ± 0.493.8 ± 0.690.7 ± 0.998.1 ± 0.571.4 ± 3.678.6 ± 4.891.0 ± 1.8

G3.1 Safeguards against answer leakage. Actually, in our implementation, reference answers never enter the sandbox. Each problem runs in a fresh directory that contains only the problem text and the skill. The container mounts only this workspace and the required dependencies, and the scorer reads the answers only after the agent has exited. External network access is disabled and boundary checks are logged. Before every batch, a canary file outside the sandbox tests the boundary; after each batch, every trace is scanned for access to answer paths and for reference values that appear without a derivation. The skill learns only from Hard100, which is disjoint from all test sets. The editing prompt also forbids hard-coded benchmark answers, although we do not rely on prompt instructions as an access-control guarantee.

G3.2 Re-run with references outside the sandbox. As eAto suggested, we re-ran the final skill with references stored outside the sandbox and audited every trace:

SubsetNL4OptNLP4LPReSoc.EasyLPCompORIndORCompLP
Original / seed-0 anchor98.093.890.898.271.478.691.0
Refs outside sandbox97.693.890.698.071.476.290.1
Traces accessing answers0/2450/1780/4030/5450/280/420/111

Conclusion: accuracy changes by at most 2.4 points, and none of the traces accessed a reference answer.

G4.1 The judge is validated against expert assessments. The human audit has two parts: 50 stratified instances across the seven subsets, used to compute κ, and all ComplexLP outputs of both OR-Skill and OptiTree, which give the accuracies in Table 9. Two OR experts checked variables and domains, every constraint, the objective, solver status and feasibility of the reported solution. GPT-5's correct counts differ from expert consensus by one (101 vs. 102 for OR-Skill; 83 vs. 82 for OptiTree).

G4.2 The evaluation tolerance is strict. Evaluation requires an absolute difference below 0.01, not a relative difference below 1%.

G4.3 Cross-judge and judge-free scoring. We also re-scored the same outputs without GPT-5: using DeepSeek-R1, from a different model family, and deterministic numerical matching at a relative tolerance of 10⁻⁶. Every run's output contract already requires a canonical objective field, so rescoring changes neither the generated model nor its solution. The table reports accuracy over the seven subsets:

ScorerOR-SkillEvoSkillOptiTreeNo skill
GPT-588.8084.6577.5075.96
DeepSeek-R188.7084.6277.4175.85
Canonical strict match88.4584.3277.1775.59

Conclusion: OR-Skill ranks first under every scorer, and the four-method ranking is unchanged. This suggests that its advantage is not specific to the GPT-5 judge.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.