acceptodds
Under review as a conference paper at ICLR 2027

OptWeaver: Learning What to Remember for Optimization Modeling

Abstract

Large language models have advanced automatic optimization modeling from natural-language descriptions. Recent knowledge-augmented methods improve performance by retrieving relevant knowledge from solved problems. However, retrieval relevance does not guarantee modeling applicability or specify how the retrieved knowledge should participate in model construction. Moreover, knowledge updates after failures may not match the localized source of the error. We propose OptWeaver, a bidirectional framework that organizes accumulated modeling knowledge into reusable skills for forward model construction and backward knowledge evolution. For each problem, OptWeaver first adapts reusable skills into temporary task-specific guidance. It then turns this guidance into concrete modeling decisions and weaves them through mathematical dependencies into a modeling graph for solver code generation. When execution fails, OptWeaver traces the same construction structure backward to localize the responsible source and guide targeted repair. Reusable principles distilled from validated repairs are incorporated into the skill system for reuse, with changes localized to relevant modeling knowledge. This design makes knowledge use explicit during model construction and aligns knowledge updates with the attributed source of modeling failures. Across six benchmarks, OptWeaver achieves an average accuracy of 78.35%, exceeding the strongest average baseline by 6.40%. The learned skills also transfer across model families, model scales and modeling frameworks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Optimization Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NEasOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

OptWeaver is a framework for optimization modeling that improves a fixed LLM through an evolving external skill library. It combines problem-family skills with shared modeling-unit knowledge, adapts them into task-specific guidance, and constructs an explicit Modeling Graph before generating solver code. Failed attempts are traced back through the construction process for targeted repair, and reusable lessons from validated corrections update the skill library during learning. The library is frozen during evaluation. Across six benchmarks, the method achieves 78.35% average accuracy, compared with 71.95% for the strongest average baseline. Additional experiments examine component ablations and skill transfer across models and frameworks.

Critical strengths
  1. Explicitly relating reusable knowledge to concrete modeling decisions and their dependencies is a useful design choice. It provides a more structured basis for diagnosing failures and deciding what knowledge to update. The directional facility-selection example illustrates how an upstream indexing error can propagate through the formulation and be repaired at its source.

  2. The method improves average accuracy across six benchmarks, with particularly substantial gains on IndustryOR. The experiments that transfer learned skills to other models and AlphaOPT further suggest that some acquired knowledge remains useful outside the original inference pipeline. This supports the practical value of externalizing reusable modeling knowledge.

Critical weaknesses
  1. The novelty beyond existing experience libraries is insufficiently established. The paper should clarify what makes its skills substantively different from prior experience representations.
  2. The contribution of initialization and skill evolution is unclear. The authors should explain how the initial skills are constructed and compare four settings: an empty library, the frozen initial library, a library evolved from the initial library, and a library evolved from an empty library. These comparisons are helpful to demonstrate the gains from initialization and skill evolution separately.
  3. Skill-retrieval quality is not quantified. The authors should report retrieval accuracy or skill applicability as the library grows, especially when skills overlap or problems involve multiple modeling structures.
  4. The rules for skill consolidation are underspecified. Although the paper mentions modification and merging, it does not clearly specify when to update an existing skill, merge related knowledge, or create a new entry. The authors should provide these criteria and report the counts and representative examples of each operation for problem-family skills and modeling-unit knowledge.

I should be willing to increase my rating if these concerns are well-addressed.

Questions for authors
  1. Are the initial skills manually written, LLM-generated, or derived from existing resources?

  2. How does test-time repair detect an incorrect formulation when the code executes successfully? Does the agent receive reference objective values or benchmark correctness labels?

6 comments

Dear Reviewer NEas,

Thank you for your kind support and valuable comments. We hope our rebuttal adequately addresses your concerns. If so, we would greatly appreciate your consideration of a higher score. Otherwise, please let us know your further concerns, and we will continue to address them.

W1 & Q1. Novelty beyond existing experience libraries and the construction of initial skills

The novelty beyond existing experience libraries is insufficiently established. The paper should clarify what makes its skills substantively different from prior experience representations.

Thank you for this important comment. The key novelty of OptWeaver is not simply representing accumulated experiences as Skills, but connecting reusable modeling knowledge to concrete mathematical decisions and their dependencies, enabling the same knowledge structure to support model construction, error attribution, and targeted knowledge evolution.

We clarify this distinction from three perspectives.

(1) Complementary modeling knowledge at different granularities.

OptWeaver organizes reusable knowledge into two components: Problem-Family Skills and a Shared Modeling Unit Skill. The former captures the formulation logic of specific problem classes, including decision variables, constraints, objectives, modeling techniques, and code assembly rules. The latter decomposes reusable knowledge into fundamental modeling units, including modeling principles, implementation patterns, error characteristics, and repair rules.

(2) From knowledge retrieval to explicit mathematical dependencies.

Retrieving relevant experience does not guarantee its correct application. OptWeaver adapts retrieved knowledge into task-specific guidance and binds it to the variables, constraints, and objectives extracted from the current problem. These decisions are then organized in a task-specific Modeling Graph that explicitly represents their mathematical dependencies. Unlike a knowledge graph connecting experience entries, this graph represents the actual mathematical structure of the current formulation.

(3) From feedback to targeted repair and knowledge evolution.

OptWeaver reuses the construction structure to trace modeling failures backward through extracted elements, task-specific guidance, mathematical dependencies, and solver implementation. It then localizes the responsible decisions and repairs affected components while preserving unaffected ones. During training, validated corrections are distilled into reusable principles and incorporated into existing skills, merged with related knowledge, or introduced as new entries.

A concrete example. Consider the directional facility-selection problem in Appendix D, which selects exactly two facilities to maximize pairwise dispersion. The coefficients are directional (e.g., d04=24d_{04}=24 and d40=25d_{40}=25). An initial formulation incorrectly uses the unordered index scope i<ji<j, omitting reverse-direction contributions. OptWeaver uses facility-selection knowledge to construct selection and pair-activation variables, while shared modeling knowledge supports the linking constraints and objective. The Modeling Graph explicitly records their dependencies. During training, backward attribution traces the objective discrepancy to the pair-indexing decision and corrects the scope to i≠ji\neq j, recovering the correct objective value of 49.

This example illustrates how OptWeaver connects reusable knowledge to concrete modeling decisions and their correction, rather than merely storing and retrieving experiences.

The initial skills are LLM-generated, not copied from existing experience libraries or manually written from scratch. We select eight representative problem families: LP, IP, VRP, Scheduling, Assignment, Knapsack, Network Flow, and Facility Location. An LLM generates their skills using a predefined four-module structure, followed by manual review and revision to ensure mathematical correctness, applicability, and organizational consistency. These skills provide a compact foundation for subsequent knowledge evolution rather than an exhaustive problem taxonomy.

W2. Contribution of skill initialization and subsequent evolution

The contribution of initialization and skill evolution is unclear. The authors should explain how the initial skills are constructed and compare an empty library, a frozen initial library, a library evolved from the initial library, and a library evolved from an empty library.

Thank you for this valuable suggestion. We conduct four controlled experiments to examine the individual contributions of skill initialization and subsequent evolution.

Initial skill construction. We initialize the library with eight representative problem families: LP, IP, VRP, Scheduling, Assignment, Knapsack, Network Flow, and Facility Location. Each skill follows a predefined four-module structure covering unit combination logic, Python code assembly rules, advanced modeling techniques, and error diagnosis. As described in our response to W1 & Q1, these skills are LLM-generated and manually reviewed to ensure mathematical correctness and consistency.

Four experimental configurations.

  • No Persistent Skills: No persistent library; Temporary Skill construction and Modeling Graph-based repair remain enabled.
  • Frozen Initial Library: Uses the eight initial skills without further evolution.
  • Evolution from Empty Library: Starts without persistent skills and learns from training problems.
  • Evolution from Initial Library: Evolves the eight initial skills using the same training problems.

Both evolution settings use the same 200 OR-Instruct training problems. We randomly select 30 problems from each of IndustryOR, MAMO ComplexLP, OptMATH, and OptiBench, yielding 120 test problems. All settings share the same backbone model, inference configuration, and repair budget. Libraries remain frozen during testing.

Table 1: Comparison of initialization and skill evolution.

ConfigurationProblem-family skillsAccuracy
No Persistent Skills066.67%
Frozen Initial Library865.83%
Evolution from Empty Library0 → 6664.17%
Evolution from Initial Library8 → 3765.83%

Table 2: Benchmark-wise accuracy (30 problems per benchmark).

BenchmarkNo SkillsFrozenEmpty EvolutionInitial Evolution
IndustryOR53.33%50.00%60.00%56.67%
MAMO ComplexLP90.00%80.00%73.33%76.67%
OptMATH56.67%60.00%46.67%63.33%
OptiBench66.67%73.33%76.67%66.67%
Overall66.67%65.83%64.17%65.83%

The results reveal three observations.

  • Task-specific modeling remains effective without persistent skills. No Persistent Skills achieves 66.67% accuracy, demonstrating that Temporary Skills, Modeling Graph construction, and backward repair provide a structured modeling capability independently of persistent knowledge retrieval.

  • Initial skills provide general modeling guidance but require task-specific adaptation. For example, the initial Scheduling skill covers operation precedence and machine-conflict constraints, yet a job-shop scheduling formulation still omits some operation pairs sharing the same machine.

  • Initialization provides a more compact foundation for knowledge evolution. Starting from eight skills, the library evolves to 37 entries, compared with 66 when starting from an empty library. Notably, Evolution from Initial Library achieves 63.33% on OptMATH, compared with 46.67% for Evolution from Empty Library, indicating the benefit of structured prior knowledge when handling multiple interacting modeling structures.

W3. Quantifying skill-retrieval quality and applicability

Skill-retrieval quality is not quantified. The authors should report retrieval accuracy or skill applicability as the library grows, particularly when skills overlap or problems involve multiple modeling structures.

Thank you for highlighting this important aspect. We further analyzed the training logs and test-time execution records to examine retrieval coverage, skill applicability, and knowledge organization as the library grows.

(1) Retrieval coverage during library evolution.

We tracked the number of problem-family skills and the corresponding retrieval behavior throughout training. The problem-family library progressively expanded from the initial eight skills to 12, 18, 27, and finally 37 skills across the recorded training stages.

The non-empty retrieval rate was 100% at all four training checkpoints. On the 120 test problems, the retrieval rate reached 98.33% (118/120), indicating that the learned library provides retrievable problem-family knowledge for nearly all unseen evaluation instances.

Importantly, this metric measures whether at least one persistent skill is retrieved, rather than whether every retrieved skill is necessarily applicable. We therefore distinguish retrieval coverage from the subsequent task-specific adaptation and mathematical dependency construction.

(2) Handling overlapping knowledge and complementary modeling structures.

OptWeaver does not automatically create a new skill for every encountered problem. Instead, it examines the relationship between newly validated modeling knowledge and existing skills, allowing complementary rules to be integrated into an existing entry when appropriate.

For example, in training problem 109, a fairness constraint was incorporated into the existing continuous_resource_allocation skill. In training problem 144, a demand-satisfaction constraint was further integrated into the same skill while preserving the previously acquired fairness-related knowledge.

These updates demonstrate that the library can accumulate complementary modeling rules within an existing problem-family representation, rather than creating separate entries for every newly encountered constraint combination.

(3) Applicability to problems involving multiple modeling structures.

Consider the satellite-selection problem encountered during training. The formulation simultaneously involves budget constraints, minimum selection requirements, and dependencies among selected satellites. Following successful modeling and validation, OptWeaver introduces the binary_selection_budget_precedence problem-family skill, which captures the combination of binary selection decisions, resource constraints, and logical dependencies.

This example demonstrates that OptWeaver can identify and retain reusable knowledge arising from the interaction of multiple modeling structures. When subsequent problems contain similar mathematical relationships, the corresponding skills can be retrieved and adapted to the current modeling objects.

Reviewer_gRc4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a way for OR skill construction, which include both foward-application and backward-fix.

Critical strengths

This paper is well-motivated, since OR problems are highly structured, store and reorg structured skills is the correct way to go. The performance improvement seems huge. The backward fix part is clean and controlled.

Critical weaknesses

The presentation is a bit confusing, I feel concrete examples attached to each part is important to make the writeup easier to follow. For example, what are some problem-family skills and what are some shared modeling unit skills. Also, how a problem solving can correct call and invoke these skills.

The test-time repair trigger is unclear, without a reference solution, how does the system know a formulation is wrong at test time?

The training data source and the evaluation sets are unspecified.

No variance reporting.

Questions for authors
  1. Can the authors add some concrete examples for each part for demonstration

  2. At test time, what signal decides that a formulation has failed and triggers repair? Is the reference objective value ever used?

  3. Where do the 400 training problems come from? What are the exact test-set sizes per benchmark?

  4. Can you report mean with std over multiple runs (for main results at least, table 1, figure 3, figure 4)?

4 comments

Dear Reviewer gRc4,

Thank you for your kind support and valuable comments. We sincerely hope our rebuttal adequately address your concerns. If so, we would deeply appreciate it if you could consider raising your score. If not, please let us know your further concerns, and we will continue actively responding to your comments.

W1 & Q1. Concrete examples of skills and their application

The distinction between problem-family skills and shared modeling unit skills, as well as how they are retrieved, combined, and applied during problem solving, needs clearer illustration.

Thank you for this helpful suggestion. We have provided a detailed facility-selection example in Appendix D to illustrate the two types of skills and how they work together during model construction and repair.

  • Problem-family skills provide high-level formulation guidance for a specific class of optimization problems. In Appendix D, the integer_facility_selection skill describes how to formulate facility-selection decisions, pairwise activation variables, associated linking constraints, and the corresponding objective structure.

  • Shared modeling unit skills provide reusable knowledge for constructing individual modeling components across different problem families. For example, the shared library contains units such as V3 (binary variables), C10 (linking constraints), and O1 (maximization objectives). Each unit provides its modeling principles, implementation patterns, common errors, and corresponding repair guidance.

  • The two types of skills are jointly activated during problem solving. For the facility-selection problem in Appendix D, OptWeaver first identifies the problem type and extracts the required modeling elements, including facility-selection variables yiy_i, pairwise activation variables zijz_{ij}, the cardinality constraint, and the maximization objective. It then retrieves the corresponding problem-family skill and relevant shared modeling units, adapting them into temporary task-specific guidance. These skills guide the construction of the Modeling Graph, which explicitly connects variables, linking constraints, and objective terms through their mathematical dependencies. Finally, the graph is translated into executable solver code.

  • The same example also demonstrates how these skills support error attribution and repair. The initial formulation incorrectly represents directional facility pairs as unordered pairs (i<ji<j), thereby omitting reverse-direction contributions from the objective. OptWeaver traces the error to the indexing scope of the pairwise variables and corrects it to i≠ji\neq j, subsequently reconstructing the affected linking constraints and objective terms. The repaired formulation obtains the correct objective value.

This example illustrates that problem-family skills provide overall formulation guidance, while shared modeling unit skills support the construction of individual mathematical components. Their combination enables OptWeaver to construct, diagnose, and repair optimization models through explicit mathematical dependencies.

W2 & Q2. Test-time repair trigger without reference solutions

The signal that triggers test-time repair is unclear, particularly when reference solutions are unavailable. It is also necessary to clarify whether reference objective values are used during inference.

Thank you for pointing out this ambiguity. Reference-guided validation is exclusively used during training and is disabled at test time. Test-time repair relies on solver execution feedback rather than access to reference solutions.

  • During testing, OptWeaver first generates executable solver code from the frozen Modeling Skill Library. The generated program is then executed by the optimization solver. Solver execution and optimization feedback provide the signal for backward attribution and local repair. When the execution or solver feedback indicates an unsuccessful formulation, OptWeaver traces the failure through the Modeling Graph and performs targeted repair, with at most three repair rounds.

  • Reference objective values are never provided to OptWeaver during test-time inference. They are used only by the benchmark evaluation procedure to determine whether the final generated formulation achieves the correct objective value. They do not participate in model generation, repair triggering, or repair decisions.

  • Test-time repair is strictly instance-specific. The system can revise the current formulation based on solver feedback, but these corrections are not written back into the persistent Modeling Skill Library. The same frozen skill library is therefore maintained throughout evaluation.

This distinction separates solver-grounded local repair at inference time from reference-guided knowledge acquisition during training, ensuring that test-time modeling does not rely on reference solutions or additional persistent learning.

W3 & Q3. Training data source and evaluation-set sizes

The source of the 400 training problems and the exact number of test instances in each evaluation benchmark are not clearly specified.

Thank you for pointing out the need for more detailed dataset information. We clarify the training data source and the exact evaluation-set sizes below.

  • Training data. The 400 training problems are randomly sampled from the OR-Instruct dataset, which contains 3,000 optimization modeling problems. These problems are used exclusively for learning and evolving the persistent Modeling Skill Library.

  • Evaluation benchmarks. We evaluate OptWeaver on six public optimization modeling benchmarks. The exact numbers of instances used in our evaluation are summarized in Table 1.

    Table 1: Statistics of the evaluation benchmarks.

    BenchmarkNumber of Test Problems
    NL4Opt245
    MAMO EasyLP642
    MAMO ComplexLP203
    IndustryOR100
    OptiBench605
    OptMATH166
    Total1,961
  • These benchmarks cover diverse optimization modeling scenarios, ranging from standard linear programming to complex constraint interactions, integer decisions, logical dependencies, and combinatorial optimization.

Thus, the 400 OR-Instruct problems are used for knowledge acquisition, while the six public benchmarks provide independent evaluation sets for measuring the modeling performance and transferability of the learned skills.

W4 & Q4. Variance across multiple runs

The experimental results lack variance reporting. Can the authors report mean and standard deviation over multiple independent runs, particularly for the main results, Table 1, Figure 3, and Figure 4?

Thank you for this suggestion. To examine the stability of OptWeaver, we conduct three independent runs and report the mean accuracy and standard deviation on the available benchmark results.

  • The results on IndustryOR, OptMATH, and MAMO ComplexLP are summarized in Table 2.

    Table 2: Accuracy across three independent runs (mean ± std).

    BenchmarkRun 1Run 2Run 3Mean ± Std
    IndustryOR70.00%63.00%65.00%66.00% ± 2.94%
    OptMATH47.60%44.58%44.57%45.58% ± 1.43%
    MAMO ComplexLP81.80%80.79%80.30%80.96% ± 0.62%
  • The results show relatively stable performance across independent runs, particularly on MAMO ComplexLP, where the standard deviation is only 0.62 percentage points.

  • We will incorporate mean and standard deviation reporting into the corresponding experimental results and figures to provide a more complete assessment of performance stability.

AuthorsOpenReview ↗

We are looking forward to your response.

Dear Reviewer gRc4,

We would like to extend our sincere gratitude for the time and effort you have devoted to reviewing our submission. Your valuable feedback, insightful comments, and constructive suggestions have been invaluable to us, guiding us in improving the quality of our work!

We eagerly await your feedback to understand if our responses have adequately addressed all your concerns. If so, we would deeply appreciate it if you could raise your score. If not, we are eager to address any additional queries you might have, which will enable us to enhance our work further.

Once again, thank you for your guidance and support.

Best,

Authors

Reviewer_mPPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OptWeaver, a knowledge-augmented framework for automatic optimization modeling. It organizes reusable knowledge into problem-family and modeling-unit skills, and introduces a bidirectional mechanism that uses a Modeling Graph for structured model construction, error attribution, and localized skill evolution. Experiments on six benchmarks show that OptWeaver achieves 78.35% average accuracy, outperforming the strongest baseline by 6.40 percentage points. Additional experiments demonstrate the transferability of learned skills across model families, scales, and frameworks.

Critical strengths
  1. Structured knowledge reuse and error-driven evolution. The paper presents a coherent framework that connects reusable modeling skills with explicit mathematical dependencies, enabling structured model construction, targeted error repair, and localized knowledge updates.
  2. Strong empirical results and knowledge transferability. OptWeaver achieves the best average accuracy across six benchmarks, with notable improvements on complex optimization tasks. The transfer experiments further demonstrate that learned skills can benefit different model families, scales, and modeling frameworks.
Critical weaknesses
  1. Limited novelty and insufficient validation of key mechanisms. The framework combines existing ideas such as skill-based memory, graph-based modeling, and iterative repair. The novelty of these components is not sufficiently clear. Moreover, the ablation studies do not fully isolate the benefits of structured error attribution and localized skill evolution from standard repair and memory-update strategies.
  2. Concerns about experimental fairness and verification reliability. OptWeaver uses 400 training problems, reference-guided learning, and multiple repair rounds, but it is unclear whether the baselines have comparable computational budgets. Additionally, solver-based verification may fail to detect semantically incorrect formulations, raising concerns about the reliability of accumulated skills.
Questions for authors
  • Q1. What is the independent contribution of backward attribution? Could the authors compare the proposed graph-based attribution with standard LLM-based iterative repair under the same solver feedback and repair budget? This would help clarify whether the gains come from structured attribution or additional repair attempts.

  • Q2. How effective is localized skill evolution compared with simpler memory-update strategies? Could the authors compare localized skill updates with directly appending successful experiences or rewriting existing skills? This would better demonstrate the value of the proposed knowledge evolution mechanism.

  • Q3. How is semantic correctness ensured during skill evolution? Solver execution success does not necessarily imply a correct mathematical formulation. How does the framework detect missing constraints or incorrect objectives and prevent such errors from contaminating the skill library?

  • Q4. Are the experimental comparisons computationally fair? OptWeaver uses 400 training problems, reference-guided learning, and multiple repair rounds. Could the authors clarify whether the baselines have comparable resources and report performance under matched token and solver-call budgets?

5 comments

Dear Reviewer mPPY,

Thank you for your kind support and valuable comments. We sincerely hope our rebuttal adequately address your concerns. If so, we would deeply appreciate it if you could consider raising your score. If not, please let us know your further concerns, and we will continue actively responding to your comments.

W1 & Q1. Independent contribution of backward attribution

The independent contribution of structured backward attribution over conventional LLM-based iterative repair remains unclear, particularly under identical solver feedback and repair budgets.

Thank you for raising this important point. To further validate the contribution of backward attribution, we conduct a controlled comparison between OptWeaver and standard LLM-based iterative repair.

  • We compare two repair strategies:

    • Standard LLM Iterative Repair: Diagnoses errors based on the original problem, current modeling artifacts, and solver feedback, and directly modifies variables, constraints, objectives, or code without explicitly tracing the earliest source of the error.
    • OptWeaver Backward Attribution: Exploits the Modeling Graph and intermediate modeling artifacts to trace errors back to their responsible modeling units, make targeted corrections, and update the affected downstream structures while preserving correct modeling decisions.
  • Both methods use identical solver feedback and a maximum of five repair attempts. We randomly select 30 problems from each of four benchmarks (IndustryOR, MAMO ComplexLP, OptMATH, and OptiBench), yielding 120 test problems. For a controlled comparison, we further select the 50 problems that initially fail validation under both methods.

  • The results are shown in Table 1. OptWeaver achieves higher cumulative repair success rates at all five repair budgets, with an advantage already evident after the first attempt.

    Table 1: Cumulative repair success rates under matched repair budgets.

    Maximum Repair AttemptsStandard RepairOptWeaver
    18/50 (16%)11/50 (22%)
    211/50 (22%)13/50 (26%)
    311/50 (22%)14/50 (28%)
    413/50 (26%)15/50 (30%)
    513/50 (26%)16/50 (32%)

    After five attempts, OptWeaver successfully repairs 16 problems, compared with 13 for standard iterative repair. This improvement under the same repair budget indicates that the gains come from more effective error localization rather than additional repair opportunities.

  • A milkshake production problem further illustrates the difference. Both methods initially treat the production quantity as a continuous variable, although it should be an integer. OptWeaver traces the error to the variable-domain definition during modeling element extraction, corrects the variable type, and preserves the other valid modeling elements. It then updates the downstream formulation and obtains the correct objective value. In contrast, standard iterative repair incorrectly attributes the unusually low production quantity to a missing minimum-production requirement and repeatedly adjusts the production lower bound. Although it eventually corrects the variable type in the fifth attempt, it retains an additional minimum-total-production constraint that is not required by the problem.

These results demonstrate that backward attribution enables more precise error localization and targeted repair, avoiding repeated trial-and-error modifications and preserving correct modeling structures.

W1 & Q2. Effectiveness of localized skill evolution

The effectiveness of localized skill evolution over simpler memory-update strategies, such as appending successful experiences or rewriting existing skills, requires further validation.

Thank you for this valuable suggestion. To directly evaluate the effectiveness of localized skill evolution, we introduce two additional baselines, Append and Rewrite, and compare them with OptWeaver in terms of modeling accuracy, knowledge-library growth, and representative cases.

  • We consider three knowledge-update strategies:

    • Append: Stores each successful experience as an independent entry and retrieves it together with the existing main skills in subsequent modeling tasks.
    • Rewrite: Incorporates new experiences by rewriting and replacing existing skills, without maintaining separate experience entries. New problem-family skills can still be created when necessary.
    • OptWeaver: Selectively applies Keep, Modify, Create, or Merge according to the structural relationship between new knowledge and existing skills.
  • All three methods use the same 200 OR-Instruct training problems, with identical base models, training procedures, and parameter configurations. For evaluation, we randomly sample 30 problems from each of four benchmarks (IndustryOR, MAMO ComplexLP, OptMATH, and OptiBench), yielding 120 test problems. The same inference settings and repair budget are used.

  • OptWeaver achieves the highest final accuracy, as shown in Table 2.

    Table 2: Performance comparison of different knowledge-update strategies.

    MethodCorrect PredictionsAccuracy
    Append72/12060.00%
    Rewrite71/12059.17%
    OptWeaver79/12065.83%

    OptWeaver improves accuracy by 5.83 percentage points over Append and 6.66 percentage points over Rewrite.

  • We further track the evolution of the three knowledge libraries throughout training. The results are presented in the newly added Figure 5.

    Table 3: Growth of problem-family skills and accumulated experiences.

    Processed ProblemsOptWeaver SkillsAppend Main SkillsAppend ExperiencesRewrite Main Skills
    08808
    50128448
    100188858
    1502781278
    2003781678

    Append continuously accumulates independent experiences, reaching 167 entries, while Rewrite maintains eight main skills through 165 rewriting operations. In contrast, OptWeaver selectively expands its problem-family skills from 8 to 37, incorporating new modeling structures into the existing knowledge organization.

  • Two representative cases further illustrate these differences:

    • Truck Allocation: Append retrieves additional experiences related to fleet optimization, resource allocation, and ratio constraints. However, these experiences fail to capture the mathematical meaning of "average net profit per truck," leading to an incorrect objective that divides total profit by the number of regions. OptWeaver preserves the correct objective semantics and obtains the correct result on the first solve.
    • Aircraft Landing Scheduling: Append retrieves experiences from fixed-order project scheduling and employee scheduling, but fails to adapt them to the conditional ordering structure of the problem. This results in an insufficient Big-M parameter and an infeasible model that remains unresolved after three repair attempts. OptWeaver correctly constructs the conditional separation constraints and obtains the correct objective value on the first solve.

These results demonstrate that effective knowledge evolution requires more than accumulating experiences or repeatedly rewriting existing skills. By selectively integrating new knowledge according to mathematical structure, OptWeaver achieves both broader skill coverage and higher modeling accuracy.

W2 & Q3. Semantic correctness and reliability of skill evolution

Solver-based validation alone may fail to identify semantically incorrect formulations, such as missing constraints or incorrect objectives. How does OptWeaver prevent such errors from contaminating the skill library?

Thank you for highlighting the distinction between numerical validation and semantic correctness. To further examine the reliability of skill evolution, we clarify the semantic analysis and backward-diagnosis mechanisms in OptWeaver and introduce an independent multi-model semantic auditing experiment.

  • During forward modeling, OptWeaver extracts decision variables, variable domains, constraints, objective functions, and their mathematical dependencies from the original problem. These elements are organized into task-specific guidance and a Modeling Graph, providing structured references for subsequent error diagnosis.

  • During backward attribution, OptWeaver uses the original problem, intermediate modeling artifacts, and diagnostic knowledge to identify potential errors, including incorrect variable definitions, missing constraints, objective-semantic mismatches, and inconsistent dependencies between modeling units. Only trajectories that pass numerical validation are eligible for subsequent knowledge extraction and persistent skill updates. Reference implementations may assist in diagnosing unresolved training problems, but corrections must still be applied to the current formulation and validated before knowledge is retained.

  • To further examine semantic reliability, we conduct an independent multi-model auditing experiment on 100 OR-Instruct training problems. We compare numerical-only validation with numerical validation augmented by semantic auditing using three independent models: DeepSeek-V3, GLM-5.2, and Qwen3.6-35B-A3B. Each auditor independently examines the original problem and the complete generated code, focusing on variable definitions, constraint completeness, objective semantics, and mathematical dependencies, without access to reference answers, solver results, or other auditors' judgments. A trajectory is eligible for persistent skill updates only if it passes numerical validation and receives positive judgments from at least two auditors.

  • The results are summarized in Table 4.

    Table 4: Numerical validation and multi-model semantic auditing.

    MetricNumerical OnlyNumerical + Audit
    Training Problems100100
    Numerical Pass Rate84.00%82.00%
    Eligible Trajectories8474
    Token Consumption11.47M16.18M
  • A further cross-analysis of the 100 candidate programs reveals the complementary roles of the two validation signals.

    Table 5: Agreement between numerical validation and semantic auditing.

    Validation OutcomeNumber of Problems
    Both accepted74
    Numerical accepted, semantic rejected or uncertain8
    Semantic accepted, numerical rejected5
    Neither accepted13
    Total100

    In particular, 8 programs pass numerical validation but are rejected or considered uncertain by semantic auditors, demonstrating that solver-based checks alone may overlook certain modeling errors. Conversely, five programs receive positive semantic judgments but fail numerical validation, confirming that semantic auditing cannot replace numerical verification.

  • We also compare the resulting Modeling Graphs. The average Jaccard similarity between their modeling-unit sets reaches 84.94%, indicating substantial consistency in the main modeling structures identified under the two validation strategies.

  • A material-mixing case further illustrates this issue. Semantic auditing identifies a potentially missing constraint requiring the material proportions to sum to one, which may not be detected through objective-value consistency alone. Meanwhile, some auditors incorrectly interpret this condition as conflicting with the proportion upper bounds specified in the problem. This example highlights the importance of examining semantic requirements while considering the complete mathematical context.

These results motivate a complementary validation design: numerical verification remains the default mechanism, while semantic analysis and backward diagnosis provide additional checks on modeling correctness. Independent multi-model auditing can further strengthen semantic screening when stricter validation is required, with its additional computational cost taken into account.

Comments on the paper

1 comment

Dear Reviewers,

Please note that the author rebuttal is now available. We kindly ask you to read it, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.

Best regards

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.