- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
OptWeaver is a framework for optimization modeling that improves a fixed LLM through an evolving external skill library. It combines problem-family skills with shared modeling-unit knowledge, adapts them into task-specific guidance, and constructs an explicit Modeling Graph before generating solver code. Failed attempts are traced back through the construction process for targeted repair, and reusable lessons from validated corrections update the skill library during learning. The library is frozen during evaluation. Across six benchmarks, the method achieves 78.35% average accuracy, compared with 71.95% for the strongest average baseline. Additional experiments examine component ablations and skill transfer across models and frameworks.
Critical strengths
-
Explicitly relating reusable knowledge to concrete modeling decisions and their dependencies is a useful design choice. It provides a more structured basis for diagnosing failures and deciding what knowledge to update. The directional facility-selection example illustrates how an upstream indexing error can propagate through the formulation and be repaired at its source.
-
The method improves average accuracy across six benchmarks, with particularly substantial gains on IndustryOR. The experiments that transfer learned skills to other models and AlphaOPT further suggest that some acquired knowledge remains useful outside the original inference pipeline. This supports the practical value of externalizing reusable modeling knowledge.
Critical weaknesses
- The novelty beyond existing experience libraries is insufficiently established. The paper should clarify what makes its skills substantively different from prior experience representations.
- The contribution of initialization and skill evolution is unclear. The authors should explain how the initial skills are constructed and compare four settings: an empty library, the frozen initial library, a library evolved from the initial library, and a library evolved from an empty library. These comparisons are helpful to demonstrate the gains from initialization and skill evolution separately.
- Skill-retrieval quality is not quantified. The authors should report retrieval accuracy or skill applicability as the library grows, especially when skills overlap or problems involve multiple modeling structures.
- The rules for skill consolidation are underspecified. Although the paper mentions modification and merging, it does not clearly specify when to update an existing skill, merge related knowledge, or create a new entry. The authors should provide these criteria and report the counts and representative examples of each operation for problem-family skills and modeling-unit knowledge.
I should be willing to increase my rating if these concerns are well-addressed.
Questions for authors
-
Are the initial skills manually written, LLM-generated, or derived from existing resources?
-
How does test-time repair detect an incorrect formulation when the code executes successfully? Does the agent receive reference objective values or benchmark correctness labels?
6 comments
W2. Contribution of skill initialization and subsequent evolution
The contribution of initialization and skill evolution is unclear. The authors should explain how the initial skills are constructed and compare an empty library, a frozen initial library, a library evolved from the initial library, and a library evolved from an empty library.
Thank you for this valuable suggestion. We conduct four controlled experiments to examine the individual contributions of skill initialization and subsequent evolution.
Initial skill construction. We initialize the library with eight representative problem families: LP, IP, VRP, Scheduling, Assignment, Knapsack, Network Flow, and Facility Location. Each skill follows a predefined four-module structure covering unit combination logic, Python code assembly rules, advanced modeling techniques, and error diagnosis. As described in our response to W1 & Q1, these skills are LLM-generated and manually reviewed to ensure mathematical correctness and consistency.
Four experimental configurations.
- No Persistent Skills: No persistent library; Temporary Skill construction and Modeling Graph-based repair remain enabled.
- Frozen Initial Library: Uses the eight initial skills without further evolution.
- Evolution from Empty Library: Starts without persistent skills and learns from training problems.
- Evolution from Initial Library: Evolves the eight initial skills using the same training problems.
Both evolution settings use the same 200 OR-Instruct training problems. We randomly select 30 problems from each of IndustryOR, MAMO ComplexLP, OptMATH, and OptiBench, yielding 120 test problems. All settings share the same backbone model, inference configuration, and repair budget. Libraries remain frozen during testing.
Table 1: Comparison of initialization and skill evolution.
| Configuration | Problem-family skills | Accuracy |
|---|---|---|
| No Persistent Skills | 0 | 66.67% |
| Frozen Initial Library | 8 | 65.83% |
| Evolution from Empty Library | 0 → 66 | 64.17% |
| Evolution from Initial Library | 8 → 37 | 65.83% |
Table 2: Benchmark-wise accuracy (30 problems per benchmark).
| Benchmark | No Skills | Frozen | Empty Evolution | Initial Evolution |
|---|---|---|---|---|
| IndustryOR | 53.33% | 50.00% | 60.00% | 56.67% |
| MAMO ComplexLP | 90.00% | 80.00% | 73.33% | 76.67% |
| OptMATH | 56.67% | 60.00% | 46.67% | 63.33% |
| OptiBench | 66.67% | 73.33% | 76.67% | 66.67% |
| Overall | 66.67% | 65.83% | 64.17% | 65.83% |
The results reveal three observations.
-
Task-specific modeling remains effective without persistent skills. No Persistent Skills achieves 66.67% accuracy, demonstrating that Temporary Skills, Modeling Graph construction, and backward repair provide a structured modeling capability independently of persistent knowledge retrieval.
-
Initial skills provide general modeling guidance but require task-specific adaptation. For example, the initial Scheduling skill covers operation precedence and machine-conflict constraints, yet a job-shop scheduling formulation still omits some operation pairs sharing the same machine.
-
Initialization provides a more compact foundation for knowledge evolution. Starting from eight skills, the library evolves to 37 entries, compared with 66 when starting from an empty library. Notably, Evolution from Initial Library achieves 63.33% on OptMATH, compared with 46.67% for Evolution from Empty Library, indicating the benefit of structured prior knowledge when handling multiple interacting modeling structures.
W3. Quantifying skill-retrieval quality and applicability
Skill-retrieval quality is not quantified. The authors should report retrieval accuracy or skill applicability as the library grows, particularly when skills overlap or problems involve multiple modeling structures.
Thank you for highlighting this important aspect. We further analyzed the training logs and test-time execution records to examine retrieval coverage, skill applicability, and knowledge organization as the library grows.
(1) Retrieval coverage during library evolution.
We tracked the number of problem-family skills and the corresponding retrieval behavior throughout training. The problem-family library progressively expanded from the initial eight skills to 12, 18, 27, and finally 37 skills across the recorded training stages.
The non-empty retrieval rate was 100% at all four training checkpoints. On the 120 test problems, the retrieval rate reached 98.33% (118/120), indicating that the learned library provides retrievable problem-family knowledge for nearly all unseen evaluation instances.
Importantly, this metric measures whether at least one persistent skill is retrieved, rather than whether every retrieved skill is necessarily applicable. We therefore distinguish retrieval coverage from the subsequent task-specific adaptation and mathematical dependency construction.
(2) Handling overlapping knowledge and complementary modeling structures.
OptWeaver does not automatically create a new skill for every encountered problem. Instead, it examines the relationship between newly validated modeling knowledge and existing skills, allowing complementary rules to be integrated into an existing entry when appropriate.
For example, in training problem 109, a fairness constraint was incorporated into the existing continuous_resource_allocation skill. In training problem 144, a demand-satisfaction constraint was further integrated into the same skill while preserving the previously acquired fairness-related knowledge.
These updates demonstrate that the library can accumulate complementary modeling rules within an existing problem-family representation, rather than creating separate entries for every newly encountered constraint combination.
(3) Applicability to problems involving multiple modeling structures.
Consider the satellite-selection problem encountered during training. The formulation simultaneously involves budget constraints, minimum selection requirements, and dependencies among selected satellites. Following successful modeling and validation, OptWeaver introduces the binary_selection_budget_precedence problem-family skill, which captures the combination of binary selection decisions, resource constraints, and logical dependencies.
This example demonstrates that OptWeaver can identify and retain reusable knowledge arising from the interaction of multiple modeling structures. When subsequent problems contain similar mathematical relationships, the corresponding skills can be retrieved and adapted to the current modeling objects.
Dear Reviewer NEas,
Thank you for your kind support and valuable comments. We hope our rebuttal adequately addresses your concerns. If so, we would greatly appreciate your consideration of a higher score. Otherwise, please let us know your further concerns, and we will continue to address them.
W1 & Q1. Novelty beyond existing experience libraries and the construction of initial skills
Thank you for this important comment. The key novelty of OptWeaver is not simply representing accumulated experiences as Skills, but connecting reusable modeling knowledge to concrete mathematical decisions and their dependencies, enabling the same knowledge structure to support model construction, error attribution, and targeted knowledge evolution.
We clarify this distinction from three perspectives.
(1) Complementary modeling knowledge at different granularities.
OptWeaver organizes reusable knowledge into two components: Problem-Family Skills and a Shared Modeling Unit Skill. The former captures the formulation logic of specific problem classes, including decision variables, constraints, objectives, modeling techniques, and code assembly rules. The latter decomposes reusable knowledge into fundamental modeling units, including modeling principles, implementation patterns, error characteristics, and repair rules.
(2) From knowledge retrieval to explicit mathematical dependencies.
Retrieving relevant experience does not guarantee its correct application. OptWeaver adapts retrieved knowledge into task-specific guidance and binds it to the variables, constraints, and objectives extracted from the current problem. These decisions are then organized in a task-specific Modeling Graph that explicitly represents their mathematical dependencies. Unlike a knowledge graph connecting experience entries, this graph represents the actual mathematical structure of the current formulation.
(3) From feedback to targeted repair and knowledge evolution.
OptWeaver reuses the construction structure to trace modeling failures backward through extracted elements, task-specific guidance, mathematical dependencies, and solver implementation. It then localizes the responsible decisions and repairs affected components while preserving unaffected ones. During training, validated corrections are distilled into reusable principles and incorporated into existing skills, merged with related knowledge, or introduced as new entries.
A concrete example. Consider the directional facility-selection problem in Appendix D, which selects exactly two facilities to maximize pairwise dispersion. The coefficients are directional (e.g., d04=24 and d40=25). An initial formulation incorrectly uses the unordered index scope i<j, omitting reverse-direction contributions. OptWeaver uses facility-selection knowledge to construct selection and pair-activation variables, while shared modeling knowledge supports the linking constraints and objective. The Modeling Graph explicitly records their dependencies. During training, backward attribution traces the objective discrepancy to the pair-indexing decision and corrects the scope to i=j, recovering the correct objective value of 49.
This example illustrates how OptWeaver connects reusable knowledge to concrete modeling decisions and their correction, rather than merely storing and retrieving experiences.