- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces OR-Skill, an approach that equips a general-purpose coding agent (Claude Code running DeepSeek V3.2) with a specialized optimization-modeling skill set consisting of a SKILL.md file, reference files, and solver scripts. The skill is refined offline via an evolutionary loop using "Hard100"—a newly curated dataset of 100 complex optimization problems with expert ground-truth models.
The evolutionary process operates in rounds:
- A parent skill is sampled from a pool based on its current score.
- The agent runs the parent skill on Hard100.
- An LLM judge categorizes failures into four distinct levels (Correct, Syntax, Improvable, Logical).
- Failure modes across levels are aggregated into a root-cause report.
- An LLM updates the skill via Git commits in reverse severity order (, termed "cascade evolution"), generating child skills.
Child skills are evaluated using accuracy minus a token-length penalty (Eq. 3). After rounds, the top-performing skill is selected and frozen.
The authors claim three main contributions:
- The skill-evolution pipeline (level-based grading, root-cause attribution, cascade editing, and length-regularized selection).
- The Hard100 benchmark.
- The open-sourced final skill.
The method is evaluated across seven subsets from six standard benchmarks (NL4Opt, NLP4LP, ReSocratic, MAMO EasyLP/ComplexLP, ComplexOR, IndustryOR), using GPT-5 to judge correctness. The authors report state-of-the-art or tied performance across all subsets—for instance, achieving 91.0% vs. 82.1% (EvoSkill) on ComplexLP and 78.6% vs. 73.8% on IndustryOR. The submission also includes ablations, a judge-human agreement study, an embedding leakage check, and case studies.
Critical strengths
-
The component ablation in Table 8 is striking: the vanilla agent achieves 56.8% on ComplexLP, whereas the fully evolved skill boosts performance to 91.0%. This shows that a well-crafted domain skill does heavy lifting on hard instances. If the protocol issues raised below can be resolved, this is a very practical takeaway—it suggests that pairing a frozen agent with an inspectable, learned skill can rival or outperform specialized, fine-tuned OR models.
-
Because the skill is managed as a Git repository, every iteration produces a readable diff. I appreciate that the paper openly displays:
- The structure of the final skill (Fig. 5).
- The step-by-step update history (Table 6).
- Detailed token counts (Table 7).
- Qualitative case studies, including an explicit breakdown of a failure case (D.5).
This transparency makes it easy to see what the agent actually learned and where it still struggles, which is vital for real-world OR deployments.
Critical weaknesses
1. Substantial unacknowledged overlap with prior work weakens the novelty claims. The core methodology leans heavily on concepts already introduced in prior literature—most notably AlphaOPT (Kong, Qu et al., arXiv:2510.18428, Oct 2025), which is missing from the references. Specifically, AlphaOPT already:
- Learns reusable optimization knowledge offline using solver feedback from failed attempts.
- Formulates library building as , which is functionally identical to the token-regularized objective in Eq. 3.
- Uses progressive disclosure mechanisms for knowledge retrieval.
- Groups failure symptoms by root cause into single diagnosed issues (directly analogous to Sec. 3.6).
- Builds a taxonomy that closely mirrors Table 5 (flow conservation, inventory balance, big-M formulation, continuous vs. discrete handling, index mismatches).
- Evaluates on the exact same benchmarks (NL4OPT, NLP4LP, IndustryOR, MAMO ComplexLP).
AlphaOPT even explicitly suggests extending its library approach into agent skills in its conclusion. Beyond AlphaOPT, several other relevant works are unmentioned:
- OptiMind (arXiv:2509.22979): Error analysis converted into reusable optimization hints.
- ReLoop (arXiv:2602.15983): Verification of silent formulation errors ( in this paper).
- NEMO (arXiv:2601.21372) and OptimAI / OR-LLM-Agent: Agentic frameworks for OR modeling.
- StepORLM (ICLR 2026): Self-evolving OR modeling.
- General skill-evolution frameworks like SkillOpt and Trace2Skill.
Once these works are accounted for, the remaining novelty is largely constrained to packaging knowledge specifically into a coding-agent skill, population-based selection, and the specific reverse-cascade editing order.
2. Mathematical inconsistencies in the selection objective and flaws in evaluation protocol.
(a) Equation 3 breaks under the reported numbers. Plugging in the given weights (, ) and the token lengths of the final skill (3,817 and 7,427 from Table 7), the length penalty evaluates to:
Since accuracy is bounded in , the final skill receives a total score around . Meanwhile, the baseline would score drastically higher due to its minimal length. Under a strict , Eq. 3 would always revert to . Furthermore, sampling proportional to scores becomes ill-defined when scores are negative. As presented, the selection math couldn't have generated the reported results, making it difficult to trust the Occam ablation.
(b) Potential evaluation artifacts inflating margins.
- Permissive error tolerances: The evaluation uses a 1% relative tolerance, whereas standard literature generally uses .
- LLM Judge Bias: The GPT-5 judge appears to favor the proposed method. Switching to exact answer extraction causes OR-Skill's score to drop by 9.0 points, compared to only 6.3 for OptiTree (Table 9).
- Inconsistent denominators in Table 1:
- On IndustryOR, agent baselines use , while training baselines use .
- On ComplexOR, 71.4% corresponds to , but EvoSkill's reported 69.1% doesn't map to any , and the reported gain is smaller than a single problem instance.
- Confounded baselines & missing controls: Fine-tuning baselines rely on a smaller backbone (Qwen3-8B), and the direct control—Claude Code + DeepSeek V3.2 without any skill—is missing from Table 1.
- Lack of variance reporting: Results are reported without seeds or standard deviations. The cascade ordering claim is evaluated against a single pass rather than a forward baseline, and on EasyLP, two ablated variants perform no better than the bare agent.
Questions for authors
- What were the exact Eq. 3 scores for and ? Was the final skill picked purely by Hard100 accuracy or by Eq. 3? How did you handle score-proportional sampling when raw scores went negative?
- Could you explicitly map out which components are novel relative to AlphaOPT (particularly Eq. 3 vs. its size-regularized objective, and Sec. 3.6 vs. its root-cause diagnosis)? Are you able to provide a direct comparison against AlphaOPT using a matched backbone?
- Can you benchmark all methods on identical instance sets using a standard tolerance and deterministic answer extraction, reporting mean std across at least 3 seeds? Could you also clarify the denominator discrepancies on IndustryOR ( vs ) and ComplexOR in Table 1?
- Can you include (a) a forward-cascade ablation () and (b) a base row for Claude Code + DeepSeek V3.2 without any skill in Table 1? How were EvoSkill and OptSkills set up in terms of data, execution budgets, and runtime environment?
- The caption in Fig. 6 notes that Hard100 counts are "pending verification," yet Sec. C.2 draws quantitative conclusions from them. Are these counts verified and finalized?
- In Table 9, how was a "human expert consensus" accuracy established for the full ComplexLP dataset if human annotation was only performed on a 50-instance subset?
Thank you for your detailed review!
Thank you for your detailed review. We address your concerns below.
W1/Q2. Novelty relative to AlphaOPT. We apologize for omitting AlphaOPT and will cite it in the revision.
We acknowledge that Eq. 3 shares AlphaOPT's performance–size regularization principle, and Sec. 3.6 shares its root-cause consolidation principle. We do not claim either as novel in isolation.
OR-Skill integrates these principles into population-based evolution of workflow instructions, domain references, and executable utilities. Its specific update mechanism combines expert-model-based L₀–L₃ grading, level-wise diagnosis, and an L₃→L₂→L₁ editing cascade.
W1.3 Comparison with AlphaOPT. We have also added an experiment comparing OR-Skill with our re-implementation of AlphaOPT:
W1.4 Originality. Our contribution is the integrated framework for severity-structured evolution of Git-versioned agent skills and its empirical benefits, rather than the independent novelty of each component. We will revise our claims accordingly.
W2.1 Eq. 3 (W2(a), Q1). See G1.1 and G1.2. s* was selected by Eq. 3, and the scores of s₀ and s* are given in our response to sHSS (Q1.2).
W2.2 Error tolerance. We apologize for the mistake: we use an absolute tolerance of 0.01, not a relative tolerance. We will correct this in the revision. We also report additional rescoring at a relative tolerance of 10⁻⁶ (G4.2, G4.3).
W2.3 LLM judge bias. We understand this concern. However, the expert audit and cross-scorer results do not indicate systematic bias in favor of OR-Skill.
GPT-5 undercounts OR-Skill and overcounts OptiTree by one instance relative to expert consensus. All five label disagreements per method are reported in sHSS Q2.5. The extractor (κ = 0.61) fails more often on OR-Skill because of output-parsing errors. Our programs follow the skill's verification step and print solver status, variable values and verification results, so a rule-based extractor may select the wrong numerical value. App. D.3, for example, prints the optimum (1,000) and then the units transported (1,840). Canonical-format numerical comparison preserves the four-method ordering (G4.3).
W2.4 Denominators, variance, and controls. G2 addresses the denominator discrepancies, variance reporting and the missing no-skill control. The backbone confound is addressed in our response to eAto (W3.1–W3.2).
W2.5 Cascade order. Results for the cascade order (mean ± sample standard deviation over three evaluation seeds):
For the random-order variant, one of the six permutations is sampled uniformly and independently for each child's cascade, using a fixed offline random-number generator. The resulting skill is evaluated with three inference seeds.
W2.6 EasyLP ablations. On EasyLP, the original single-run result for the bare agent in Table 8 is 92.1, and the new three-seed mean is 92.2. Without the cascade or the Occam term, most of the EasyLP gain disappears while much of the ComplexLP gain remains, consistent with their role in reducing interference on easy tasks.
Q4. Setup of EvoSkill and OptSkills. Q4(a) and (b) are answered in W2.5 and G2.2. EvoSkill and OptSkills are re-run from their official implementations with DeepSeek-V3.2 and Hard100, using the same sandbox, tools, solvers, judge and held-out instance sets. The shared learning cap is 7,200 task executions and up to 7,200 associated feedback judgments per method, excluding initialization and retries, with test-set feedback forbidden. For OR-Skill this is 12 rounds × (1 parent + 5 children) × 100 tasks; its additional 36 attribution and 180 editing calls are reported separately. Matching task executions does not equalize all LLM calls, tokens or wall-clock cost.
Q5. Fig. 6 counts. See our response to eAto (W2(iii)).
Q6. Human consensus on full ComplexLP. See G4.1.
Ethics. We will credit AlphaOPT for the overlapping principles and clarify our novelty claims. Fig. 6 is recomputed from the Hard100 judgment records, and Sec. C.2 is updated accordingly.
We hope this addresses your concerns.