acceptodds
Under review as a conference paper at ICLR 2027

OPT-Zero: Data-Free Fine-Tuning of Language Models for Optimization Modeling

Abstract

Optimization modeling is central to many decision-making scenarios, but traditionally requires extensive domain expertise. While Large Language Models (LLMs) have shown promise in automating this process, current training paradigms mainly rely on human-annotated or teacher-generated datasets. This dependence introduces a Generalization Ceiling, where models overfit to narrow data distributions, and Capability Anchoring, where models' reasoning is bounded by annotator proficiency and teacher model capability. In response, we propose OPT-Zero, the first fully self-play training framework for optimization modeling that requires zero external training data. OPT-Zero employs a single LLM in a dual-role closed loop: a Proposer that synthesizes increasingly challenging optimization problems alongside their mathematical formulations and solving code, and a Solver that attempts to resolve the problems given only natural-language problem descriptions. Grounded in execution feedback from external optimization solvers, we alternately train both roles using reinforcement learning. This process fosters an auto-curriculum in which the Proposer and Solver co-evolve: generating harder valid problems by the Proposer seamlessly enhances the structural reasoning ability of the Solver. Extensive results indicate that with zero curated data, OPT-Zero matches state-of-the-art data-dependent methods while exhibiting substantially stronger generalizability, establishing self-play training as a highly scalable paradigm for advancing LLM reasoning in modeling and solving optimization problems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Models, Optimization Modeling, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_aL5JOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OPT-ZERO, a self-play framework for training LLMs to formulate and solve optimizations problems given NL descriptions, without needing external training data. The main idea is to jointly train two roles: (i) The proposer, which generates an NL description together with mathematical formulation and executable code; and (ii) The solver, which receives only NL descriptions and outputs the formulation and the code. Both roles are trained using GRPO, with rewards derived from parsing, canonical reconstruction, solving the declared formulation, code execution and objective value matching. In particular, after the proposer training, candidate problems are evaluated and those that produce 'mixed outcomes' are placed in a reference buffer and used for solver training. Using Qwen3-8B-BASE as the base model, the system is trained for multiple iterations. Evaluation on multiple benchmarks show substantial improvements over the base model, and comparable results with fine-tuned models for optimization. Ablation studies confirm that all ingredients of the system (i.e., solver training, proposer training, reference buffer, reward structures) contribute to the improved accuracy.

Critical strengths

Thank you for submitting your work to ICLR. These are the main strengths of the paper:

  1. Self-play approach for optimization modeling which does not rely on external data. Having worked in this space, training data for optimization modeling is indeed a huge issue. I generally like the idea of fine-tuning a model to generate high-quality data.
  2. The dual-role learning architecture. The proposer and solver are not trained as unrelated components, but rather two roles of the same shared-parameter model, trained alternately with role-specific prompts and rewards. Additional elegant features of the architecture include: using solver performance to filter and categorize the training instances and utilizing the reference buffer for subsequent training.
Critical weaknesses
  1. Methodological issues with the reward structure. While the system does assess solver-relative difficulty, difficulty is not part of the immediate proposer GRPO reward structure. On a related note, it's not clear to me how the system rewards for the diversity of the training examples. More importantly, as far as I can tell, the reward does not directly verify that the natural language description and declared formulation represent the same problem. (there are other checks such as canonical verification and code verification, but they do not guarantee equivalence).
  2. The empirical results are not particularly strong. While OPT-ZERO substantially improves its base model, it does not outperform the strongest specialized baseline (StepORLM). Critically, the evaluation also omits relevant small models such as Qwen3.5-9B and GPT-OSS-20B, which I believe should do quite good on the benchmarks. On a related note, performance on the harder benchmarks ( (OptMATH-Bench, MIPLIB-NL) is still fairly weak.

** Update after rebuttal: The authors have clarified how they framework ensures that the instances are diverse and complex enough (addressing weakness 1 above). They have also added experiments that show the merits of their framework on more powerful small models (addressing weakness 2 above). I am thus happy to raise my score to weak accept.

Questions for authors
  1. Please elaborate in detail how semantic consistency between the natural-language description and the declared formulation is verified. Even if the verification is currently partial, please suggest how it can be strengthened.
  2. How does the system ensure that the training set is diverse enough, e.g., covers a large amount of problem classes?
  3. Would the authors be able to compare against stronger SLMs such as Qwen3.5-9B and GPT-OSS-20B? As a stretch goal (perhaps for the future), it'll be great to see whether fine-tuned versions of these models could improve with the data generated by OPT-ZERO.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer aL5J (1/3)

Dear Reviewer aL5J,

We thank the reviewer for acknowledging the motivation of the study and the dual-role learning architecture of the self-play methodology. We address the raised concerns in a point-to-point way below.

[W1.1, Q2] Difficulty and diversity of the training set.

Thank you for raising these questions. Our framework incorporates difficulty adaptation and diversity control at complementary stages of the training loop.

Difficulty influences both Proposer learning and curriculum selection. Although the empirical solve rate is not used directly as the immediate Proposer reward, it determines the adaptive targets of the structural reward. Specifically, the targets for variable count, constraint count, and coupling density are updated between iterations using problems of medium Solver-relative difficulty (Section 3.2 and Appendix C), making the GRPO reward space adapted by the difficulty. Subsequent Proposer updates therefore receive structural feedback calibrated to the Solver’s current ability. Furthermore, Solve-rate estimates also guide task admission and difficulty-aware sampling (Section 3.4). This design incorporates Solver feedback without requiring multiple additional Solver rollouts for every completion during Proposer GRPO.

Our experiments support these design choices. Replacing the structured Proposer reward with a binary reward reduces Micro accuracy from 90.3% to 86.9% (Table 3). Moreover, Figure 7 shows that prioritizing harder problems consistently outperforms easy-first and medium-first sampling across all six benchmarks, with particularly noticeable gains on ComplexLP and IndOR.

Diversity is promoted through generation and buffer management, without a separate diversity reward term. Three mechanisms contribute:

Reference-conditioned generation: The Proposer prompt explicitly requests objectives and constraints that differ from the sampled reference model, together with a different application context for the story (Appendix B). Sampling different references encourages variation in both formulation and context.

Buffer-level diversity control: The reference buffer caps the number of problems sharing the same structural signature, comprising the LP/IP/MILP family, variable count, constraint count, and optimization sense (Section 3.4). Therefore, the buffer also prohibits the existence of many problems with similar structures.

Replay: We draw 20% of Solver training samples from the buffer, maintaining exposure to previously generated problems as new tasks enter the curriculum. Since the generated problems are random, the new tasks additionally inject diversity.

Appendix F.13 provides quantitative evidence for the resulting textual diversity. Among 300 retained buffer problems sampled across iterations, the Vendi Score is 244 and nearest-neighbor redundancy is 0.32, compared with 178 and 0.56 for 300 OptMATH-Train problems under the same TF-IDF analysis (Table 16). These results show greater textual diversity and less repetitive wording in the sampled training problems.

AuthorsOpenReview ↗

Reply to Reviewer aL5J (2/3)

[W1.2, Q1] Semantic consistency between problem descriptions and formulations.

Thank you for raising this question. We clarify that our current verification pipeline combines three complementary mechanisms:

Generation requirements: The Proposer is instructed to faithfully express the declared formulation in the story, including all coefficients, constants, constraints, and variable bounds, without hidden parameters. This explicitly requires the LLM to generate consistent content.

Formal verification: The canonical verifier first checks whether the declared formulation is feasible and bounded, and solves it to obtain a reference optimum. The Proposer's code is then independently executed and checked against this value. These checks establish a reliable reference for subsequent story-based reconstruction, which helps assess whether the natural-language description can lead to a solution consistent with the declared formulation.

Story-only reconstruction: Before buffer admission, the Solver makes eight attempts using only the natural-language description. We retain problems with both objective-matching and non-matching outcomes, requiring at least one successful reconstruction from the story. During training, the reconstructed formulations and code are also evaluated against the reference objective. This helps ensure the consistency between objective and model generation.

These mechanisms provide practical consistency checks without requiring external supervision. However, objective agreement does not guarantee full semantic equivalence. We deliberately rely on automatically verifiable feedback rather than introducing a teacher model or human annotations into the training loop. In this way, we avoid hacking consistency by checking only the objective agreement.

To further assess semantic reliability, we conducted an human-expert audit of 50 retained training instances during Qwen3.5-9B self-play. We found that 74% exhibited full story–formulation consistency, while another 18% contained differences in the feasible region but preserved the correct optimal objective value. The remaining 8% involved changes to optimal solutions or incomplete descriptions. Thus, most of the audited instances either maintained full semantic consistency or preserved the correct objective target used for reward computation. These findings provide additional evidence for the reliability of the objective-based supervision while identifying remaining semantic discrepancies.

Further improvements. Recent research offers several promising directions for strengthening semantic verification. First, an independent LLM-based auditor could assess story–formulation consistency at the level of variables, objectives, and individual constraints, following approaches such as OptArgus [1] and Opt-Verifier [2]. Second, semantic alignment and correction mechanisms, such as SAC-Opt [3] and SemOPT [4], could provide more fine-grained feedback to detect and correct inconsistencies that objective matching alone cannot identify. While it is not in the scope of our current work, exploring these approaches as additional verification or reward mechanisms is a promising direction for future work that combines with self-play training. We have put this as a future work in Conclusion to highlight it as an effective way to enhance the self-play framework.

[1] Li et al. (2026). OptArgus: A Multi-Agent System to Detect Hallucinations in LLM-based Optimization Modeling. arXiv:2605.11738.

[2] Liu et al. (2026). Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification. ICML 2026.

[3] Zhang et al. (2026). SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modeling. ICML 2026.

[4] Zhou et al. (2026). SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search. Findings of EMNLP 2026.

AuthorsOpenReview ↗

Reply to Reviewer aL5J (3/3)

[W2, Q3] Model performance and training of more SLMs.

Regarding model performance: We believe the empirical contribution of our method lies in achieving competitive performance without externally curated optimization training data, together with improvements on harder external tasks. In summary, we already achieved:

1. State-of-the-art accuracy on individual benchmarks and in the data-augmented setting, achieving the highest accuracy among compared fine-tuned models on IndustryOR and ReSocratic (two harder benchmarks), as well as the highest Micro-average accuracy (91.3%) over all benchmarks with OPT-Zero-D.

2. Stronger zero-shot generalization across unseen solver interfaces (CPLEX, SCIP, and Pyomo), consistently outperforming specialized baselines and the initial backbone across all evaluated benchmarks.

3. Greater independence from external training data, achieving competitive performance through self-generated problems and solver-grounded feedback, without human-annotated training instances or teacher-generated solutions.

Comparison with StepORLM. OPT-Zero-8B achieves 90.3% Micro accuracy versus 90.4% for StepORLM. However, this comparison also involves substantially different training-data requirements: StepORLM uses 50K verified warm-up training instances, whereas OPT-Zero generates its optimization training problems through self-play,without an external task corpus or teacher-generated solutions. Reaching this level of performance demonstrates that self-generated supervision can support a competitive specialist model. Additionally, we want to highlight that when external data are incorporated into 30% of Solver training samples, the separate OPT-Zero-D variant achieves 91.3% Micro accuracy, significantly exceeding StepORLM. In this case, we saved 50K*70% instances and their heavy preprocessing work but achieved the strongest results.

Performance on harder benchmarks. OPT-Zero improves Gemma4-31B-IT from 41.0% to 48.2% on OptMATH-Bench and from 8.92% to 11.74% on MIPLIB-NL (Table 12). MIPLIB-NL derives from industrial-scale MILPs in MIPLIB and requires translating complex model structures and constraint relationships into executable formulations. The low absolute accuracy highlights a remaining gap between the model’s current capabilities and reliable industrial-scale optimization modeling. We observe a fact that most models cannot gain high accuracy rates in these two benchmarks, implying the difficulty of them for current LLMs.

Nevertheless, our improvements on both benchmarks show that self-play advances the capabilities of LLMs more significantly, compared to all the other fine-tuned baselines.

Training of more SLMs. We first evaluate the performance of GPT-OSS-20B and Qwen3.5-9B, and the results show that our trained 8B model (90.3 Micro Avg.) consistently outperforms GPT-OSS-20B (86.9) and Qwen3.5-9B (52.7). Following the suggestion of the reviewer, we also fine-tuned GPT-OSS-20B and Qwen3.5-9B by OPT-Zero, with results summarized below (and also included in Table 4 of the revised paper).

Model / SettingNL4OptEasyLPComplexLPNLP4LPIndORReSocraticMacro Avg.Micro Avg.
Qwen3.5-9B34.770.336.936.050.050.646.452.7
+ OPT-Zero93.089.764.996.157.191.182.088.5
Absolute gain+58.2+19.4+27.9+60.1+7.1+40.4+35.5+35.8
GPT-OSS-20B91.193.055.991.052.486.678.386.9
+ OPT-Zero94.496.062.2100.059.594.884.592.4
Absolute gain+3.3+2.9+6.3+9.0+7.1+8.2+6.1+5.5

These new results again verify that our self-play training method can generalize to more SLMs, although they have different initialization, training, and model sizes.

Reviewer_U8b5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OPT-ZERO, a self-play framework for training language models for optimization modeling without an external optimization training corpus, annotated solutions, or teacher-generated demonstrations. A single model alternates between two roles: a Proposer that generates natural-language problems, mathematical formulations, and executable implementations; and a Solver that reconstructs formulations and code from the descriptions alone.

Both roles are trained using a GRPO variant. Canonical optimization solving and code execution provide rewards, while structural rewards, difficulty estimation, and a reference buffer guide curriculum construction. Starting from Qwen3-8B-Base, OPT-ZERO achieves 81.6% macro and 90.3% micro objective-value-based pass@1 across six benchmarks. Additional experiments examine component ablations, different backbones, solver-interface transfer, harder external benchmarks, and inference-time scaling.

Critical strengths
  • The paper evaluates six established benchmarks and includes meaningful ablations of Proposer training, Solver training, the reference buffer, and reward design.

  • Removing Proposer training reduces macro accuracy from 81.6% to 75.1%, while removing the buffer reduces it to 72.9%. I

Critical weaknesses
  • Proposer training also removes updates to the shared model, including formulation-to-code learning, so this ablation changes both task generation and direct optimization of the eventual Solver.

  • What proportion of retained generated problems have descriptions that faithfully preserve all constraints, variable domains, and objectives of their associated formulations?

Questions for authors

See weakness

besides,

  • Can the authors report the untuned Qwen3-8B-Base on the general reasoning tasks in Table 8 and the zero-shot solver-transfer setting in Table 9? These controls are needed to distinguish retained or improved capabilities from those already present in the initialization.
1 comment

Dear Reviewer U8b5,

We thank the reviewer for recognizing the good performance of our method and the value of our ablation studies. We address the reviewer's concerns below.

[W1] On Proposer training ablation.

Thank you for highlighting this distinction. We agree that removing Proposer training affects both task generation and updates to the shared model. Both are intended functions of the Proposer in our framework: as described in Section 3, it generates training problems while developing formulation and implementation skills that are also used by the Solver. The ablation therefore evaluates the overall contribution of this training stage.

Removing Proposer training reduces Macro/Micro accuracy from 81.6/90.3 to 75.1/84.3 (Table 3), showing that including this stage benefits the final Solver. Appendix F.12 provides complementary evidence for the Proposer’s implementation component: removing its code output and code-correctness reward reduces Micro accuracy from 90.3% to 87.1%, while retaining its formulation and story generation and the original Solver training. Together, these results support the contribution of Proposer training within the shared-model framework. We interpret the Table 3 gain as its combined contribution, without attributing it exclusively to improvements in the generated tasks.

[W2] Problem description-formulation consistency.

Thank you for raising this question. To quantify story–formulation consistency, we conducted an expert audit of retained instances that entered policy updates during Qwen3.5-9B self-play training, examining whether their descriptions faithfully preserve the constraints, variable domains, and objectives of the corresponding formulations.

The audit found that 74% of the instances were fully consistent, while another 18% contained differences in the feasible region but preserved the correct optimal objective value. For these cases, a solution faithful to the story would still receive the correct objective-matching reward. The remaining 8% involved changes to optimal solutions or incomplete descriptions. Thus, while the strict semantic consistency rate was 74%, most of the audited instances either maintained full consistency or preserved the correct reference objective used for training.

Our framework is deliberately designed to learn from self-generated problems without requiring costly external annotations. Story-only pre-scoring excludes problems with no observed objective-matching solution, while DAPO filters rollout groups without relative reward differences. These mechanisms help select useful training signals despite imperfections in generated descriptions. Together with the consistent improvements on multiple benchmarks, the audit results support the practical effectiveness of this training strategy, while highlighting opportunities for further improving semantic consistency.

[Q1] Untuned Qwen3-8B-Base performance in Table 8/9.

We thank the reviewer for this suggestion. We have added the untuned Qwen3-8B-Base to Tables 8 and 9, providing direct comparisons with OPT-Zero’s initialization. On general reasoning and coding, OPT-Zero improves GSM8K, GPQA, HumanEval, and MBPP by 1.3, 4.5, 3.1, and 6.2 percentage points, respectively. For zero-shot transfer to CPLEX, SCIP, and Pyomo, it improves all 18 benchmark–interface combinations, with gains of 12.6–51.6 percentage points, without target-interface fine-tuning. These comparisons show that performance improves over the initialization on all four reported general benchmarks and in end-to-end solver-interface transfer under the reported evaluation protocol.

SettingBenchmark / metricQwen3-8B-BaseOPT-Zero-8BGain (pp)
General reasoningGSM8K Acc.87.088.3+1.3
General reasoningGPQA Acc.35.439.9+4.5
General codingHumanEval Pass@177.480.5+3.1
General codingMBPP Pass@163.469.6+6.2
CPLEX transferNL4Opt48.892.5+43.7
CPLEX transferEasyLP57.394.5+37.2
CPLEX transferComplexLP18.934.2+15.3
CPLEX transferNLP4LP43.394.9+51.6
CPLEX transferIndOR31.057.1+26.1
CPLEX transferReSocratic38.280.1+41.9
SCIP transferNL4Opt60.692.0+31.4
SCIP transferEasyLP80.696.1+15.5
SCIP transferComplexLP24.355.9+31.6
SCIP transferNLP4LP64.097.2+33.2
SCIP transferIndOR28.847.6+18.8
SCIP transferReSocratic43.980.9+37.0
Pyomo transferNL4Opt77.590.1+12.6
Pyomo transferEasyLP73.893.6+19.8
Pyomo transferComplexLP26.159.5+33.4
Pyomo transferNLP4LP74.294.4+20.2
Pyomo transferIndOR26.247.6+21.4
Pyomo transferReSocratic65.078.2+13.2
Reviewer_kzvmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OPT-ZERO, a self-play framework for optimization modeling that requires zero external training data. The same language model alternates between the roles of "problem poser" and "problem solver": the former generates natural language problems, mathematical models, and solving code, while the latter reconstructs the model and writes code based only on the problem description; the two use execution feedback from the optimization solver for reinforcement learning, gradually increasing problem difficulty and solving capability. The authors report on six benchmarks that OPT-ZERO based on Qwen3-8B, without human annotation or teacher-generated training data, achieves overall performance close to stronger data-dependent methods.

Critical strengths
  • Shared-parameter dual-role self-play. The same model alternately acts as Proposer and Solver, forming a closed loop between problem generation and modeling/solving rather than relying on a fixed synthetic problem bank.

  • Solver-based executable feedback. The method separately validates the mathematical model and the solving code, and provides training signals through objective-value matching, which is closer to optimization modeling tasks than merely checking the form of generated text.

  • Adaptive curriculum oriented toward capability boundaries. The Proposer's reward not only requires the problem to be valid but also considers structural features such as constraint coupling, and adjusts its target according to the Solver's current capability, helping continuously generate problems with training value.

Critical weaknesses

The semantic reliability of self-generated training samples lacks direct verification. Current verification mainly ensures that the formalized model is solvable and that the objective values are consistent. Although prompt constraints and problem-statement-based reconstruction provide indirect safeguards, there is still a lack of independent auditing to quantify the inconsistency between natural language descriptions and the model, as well as its impact on training.

The independent advantage of structural rewards relative to success-rate supervision has not yet been sufficiently isolated. Existing ablations support the overall value of fine-grained Proposer rewards, but the key comparison simultaneously changes the reward and the optimization algorithm. A controlled comparison under the same algorithm and comparable compute budgets is needed to more clearly attribute the effect to the structural reward itself.

Questions for authors

see the above weakness

1 comment

Dear Reviewer kzvm,

We sincerely appreciate your positive assessment, and also thank you for acknowledging the dual-role self-play method, the solver-based execution framework, and the adaptive curriculum of OPT-Zero. We address your two concerns below.

[W1] Semantic reliability of self-generated training samples

We deliberately use two complementary filters to support training on self-generated data without requiring costly external supervision. First, before admitting a problem to the buffer, the Solver makes eight attempts using only its natural-language description. We retain the problem only if some attempts match the reference objective and others do not. This requires an observed objective match from the story alone, while keeping problems that still challenge the Solver. Second, during training, DAPO skips response groups with identical total rewards, so updates are based on comparisons between responses with different rewards. Together, these mechanisms use feedback already available within the training loop to select both training problems and response groups that provide a relative reward signal.

To directly quantify the inconsistency between natural language descriptions and the model, we conducted an expert audit of instances that entered policy updates during Qwen3.5-9B self-play training. We found a full story–formulation consistency in 74% of the audited instances. Another 18% contained discrepancies in the feasible region but preserved the optimal objective value implied by the story. In these cases, a faithful solution to the story would still match the reference objective used for reward computation. The remaining 8% involved changes to optimal solutions or incomplete descriptions. Overall, most inspected training samples supplied an appropriate reference value for our objective-based rewards.

We acknowledge that hallucinations and semantic inconsistencies remain open challenges in LLM-based optimization modeling, with erroneous instances documented across established benchmarks [1] and recent work specifically addressing the detection and correction of such errors [2, 3]. Our framework combines canonical verification, story-only buffer admission, and DAPO-based filtering to select training tasks and informative reward comparisons without external annotations or teacher supervision. The audit findings and improvements on external benchmarks together support the practical effectiveness of this self-play training strategy. However, completely eliminating the hallucinations still represents a challenge in optimization modelling. Further reducing hallucinations and strengthening story–formulation consistency remain important directions for future work.

[1] Xiao, Z., et al. (2025). A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions. IJCAI 2025.

[2] Li, Z., et al. (2026). OptArgus: A Multi-Agent System to Detect Hallucinations in LLM-based Optimization Modeling. arXiv:2605.11738.

[3] Zhou, Z., et al. (2026). SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search. Findings of EMNLP 2026.

[W2] Structural rewards vs. success-rate supervision

We thank the reviewer for this constructive suggestion. To better isolate the contribution of structural rewards, we introduce an additional controlled baseline that replaces the Proposer's structural reward with a solve-rate-based reward under the same GRPO algorithm. Specifically, for each proposed problem, we estimate its empirical solve rate (\hat{\rho}) using eight Solver rollouts from the current policy. The reward is defined as

Rsolve=(1−ρ^) I[ρ^>0].R_{\mathrm{solve}} = (1-\hat{\rho})\,\mathbb{I}[\hat{\rho}>0].

This design rewards challenging but solvable problems while assigning zero reward to problems with no observed successful solutions. Unlike the previous solve-rate-based DPO comparison (Appendix F.7), both variants now use GRPO, allowing a more direct assessment of the reward design.

Computational cost. The solve-rate reward requires 8 additional Solver rollouts for each valid Proposer-generated problem during training, introducing substantial inference overhead compared with our structural reward, which is computed directly from the declared formulation.

Due to the high computational cost of the training setting, the model only finishes its 5th self-play training round for now. We report the intermediate results here, and we promise to incorporate full results in the final manuscript.

Performance of the solve-rate-reward GRPO baseline (Pass@1).

BenchmarkCorrect / TotalAccuracy (%)
NL4Opt170 / 21379.81
MAMO-EasyLP500 / 54591.74
MAMO-ComplexLP34 / 11130.63
NLP4LP143 / 17880.34
IndustryOR20 / 4247.62
ReSocratic323 / 40380.15
Macro Avg.—68.38
Micro Avg.1190 / 149279.76
Reviewer_VdfxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OPT-Zero trains one language model in alternating Proposer and Solver roles. The Proposer generates an optimization formulation, a natural-language description, and solver code. A canonical solver supplies objective-value supervision, while the Solver reconstructs the formulation and code from the description alone. Structural rewards and a difficulty-aware reference buffer guide the curriculum. The evaluation covers six modeling benchmarks, additional backbones, and transfer to other solver interfaces.

Critical strengths
  1. The optimization-specific parser, structural rewards, and shared Proposer/Solver training turn self-play into an executable learning procedure. The two roles provide complementary practice in generating valid formulations and reconstructing them from descriptions.

  2. The before/after backbone comparisons, role ablations, and additional OptMATH/MIPLIB-NL evaluation provide substantial evidence of learning from the generated curriculum. Accounting for curriculum construction in training cost also makes the practical contribution easier to assess.

Critical weaknesses
  1. Canonical verification does not establish story–formulation consistency. Section 3.1 and Appendix A solve the generated formal fields and check whether the generated code reproduces the same objective value. Neither check independently establishes that the natural-language story specifies that model. Appendix G, Q3 addresses this concern through prompting and story-based reconstruction, but the shared model can reproduce a common interpretation error, and distinct formulations can have the same optimum on one instance. Retaining problems with mixed Solver outcomes therefore does not distinguish challenging valid tasks from incompletely specified ones. The external benchmark gains remain meaningful, but the curriculum should not be described as semantically verified without an independent audit of retained story–model pairs.

  2. The generalization and capability-preservation claims need more direct comparisons. Table 9 provides striking transfer results, but compares models with different initializations, training corpora, and training interfaces. Several baselines score approximately zero on new interfaces, making it important to distinguish API or prompt compatibility failures from failures to formulate the optimization problem. A before/after comparison using the same starting backbone would support the proposed explanation more directly. Likewise, Table 8 compares already fine-tuned systems but omits OPT-Zero's initial backbone. It supports strong general-purpose performance relative to those systems, rather than the claim in Section 4.1 that optimization gains do not come at the cost of the model's existing general capabilities.

Questions for authors
  1. Is an independent semantic audit available for retained story–model pairs, distinguishing valid difficult tasks from inconsistently specified ones?

  2. How does the same starting backbone perform on the new solver interfaces and general-purpose benchmarks, and what causes the near-zero cross-interface baseline scores?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer Vdfx (1/2)

Dear Reviewer Vdfx,

We sincerely thank you for acknowledging the methodology and substantial evaluation results of the paper. We also appreciate your support for accepting the paper. We address the concerns below.

[W1, Q1] Story–formulation consistency and canonical verification.

We thank the reviewer for highlighting this distinction. In Section 3.1, we originally described the canonical optimum as “the ground-truth objective for the given problem specified by natural language.” We have revised this to “the ground-truth objective for the declared mathematical model ((\sigma, f, V, C)).” This makes clear that canonical verification establishes the reference optimum of the declared formulation.

To directly assess the samples used for learning, we conducted an expert audit of story–formulation pairs that entered policy updates during Qwen3.5-9B self-play training. Experts inspected the natural-language descriptions and their corresponding formulations. The audit found:

74% exhibited the full story–formulation consistency.

18% contained differences in the feasible region but preserved the optimal objective value implied by the story.

8% involved changes to optimal solutions or incomplete descriptions.

These findings distinguish fully aligned retained tasks from those with specification discrepancies. For the 18% with differences in the feasible region, a faithful solution to the story would still match the reference objective used for reward computation. Overall, most of the audited instances were either fully aligned or preserved the correct objective target.

Our training pipeline combines this objective-based verification with two internal filtering mechanisms, without requiring external annotations or teacher supervision. Before buffer admission, the Solver makes eight attempts using only the story, and a task is retained only if both objective-matching and non-matching outputs are observed. During training, DAPO excludes rollout groups with identical total rewards, allowing GRPO to learn from comparisons between responses with different formulation-and-code rewards. These mechanisms determine which tasks and response groups contribute to learning.

The audit findings and improvements on external benchmarks together support the practical effectiveness of our self-play training strategy. Further reducing hallucinations and strengthening story–formulation consistency remain important directions for future work.

[W2.1, Q2] Generalization for baselines with the same starting backbone.

We thank the reviewer for the suggestion. We agree that training baseline methods with the same backbone can help clarify the sources of generalization. To this end, we train controlled adaptations of ORLM and LLMOPT from Qwen3-8B-Base, the backbone used to initialize OPT-Zero. Both adaptations use the same set of 2,631 optimization problems, full-parameter fine-tuning, and an 8,192-token context. ORLM is trained on COPT-formatted examples, while LLMOPT uses Pyomo-formatted supervised data followed by KTO on execution-verified samples generated by its own supervised model. This setup controls the initial backbone and source problems across the two baselines while retaining their method-specific training interfaces and objectives.

The revised Table 9 shows that OPT-Zero outperforms both ORLM-8B and LLMOPT-8B across all six benchmarks on CPLEX and SCIP, with all three models sharing the same initialization. Both adapted baselines score zero in these configurations, indicating that the observed transfer advantage persists after matching the initial backbone. We additionally include Qwen3-8B-Base itself to provide the requested before-and-after comparison: following Gurobi-based training, OPT-Zero improves over its initialization in all 18 benchmark–interface combinations across CPLEX, SCIP, and Pyomo, with gains of 12.6–51.6 percentage points, without target-interface fine-tuning.

Inspection of the adapted baselines' generated programs reveals syntax errors and incompatible API usage, including mixed constructs from different modeling interfaces. For example, on the 1,492-instance main evaluation set, LLMOPT-8B produces 1,286 SCIP programs with syntax errors (86.2%), including function calls incorrectly embedded in import statements. Its CPLEX outputs include 20 programs that reference an undefined Var symbol and 10 that attempt to import the Pyomo construct ConcreteModel from docplex.mp.model. An additional 1,448 CPLEX programs and 190 SCIP programs are rejected before execution by the evaluator's Pyomo-backend restriction on SolverFactory calls.

AuthorsOpenReview ↗

Reply to Reviewer Vdfx (2/2)

[W2.2] Table 8 results of OPT-Zero's initial backbone.

Thank you for pointing out the missing backbone comparison. We agree that comparison with the initial model is necessary to assess whether its existing capabilities are preserved. We now include Qwen3-8B-Base, the backbone used to initialize OPT-Zero, in the expanded comparison. Relative to this backbone, OPT-Zero improves GSM8K from 87.0 to 88.3, GPQA from 35.4 to 39.9, HumanEval from 77.4 to 80.5, and MBPP from 63.4 to 69.6. Performance improves across all four benchmarks, covering both general reasoning and code generation. This direct comparison supplies the missing evidence for the statement in Section 4.1: on the evaluated benchmarks, OPT-Zero’s gains in optimization modeling are accompanied by improvements in the backbone’s general reasoning and coding performance. We have also updated Appendix F.3 in the paper to include these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.