- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces OPT-ZERO, a self-play framework for training LLMs to formulate and solve optimizations problems given NL descriptions, without needing external training data. The main idea is to jointly train two roles: (i) The proposer, which generates an NL description together with mathematical formulation and executable code; and (ii) The solver, which receives only NL descriptions and outputs the formulation and the code. Both roles are trained using GRPO, with rewards derived from parsing, canonical reconstruction, solving the declared formulation, code execution and objective value matching. In particular, after the proposer training, candidate problems are evaluated and those that produce 'mixed outcomes' are placed in a reference buffer and used for solver training. Using Qwen3-8B-BASE as the base model, the system is trained for multiple iterations. Evaluation on multiple benchmarks show substantial improvements over the base model, and comparable results with fine-tuned models for optimization. Ablation studies confirm that all ingredients of the system (i.e., solver training, proposer training, reference buffer, reward structures) contribute to the improved accuracy.
Critical strengths
Thank you for submitting your work to ICLR. These are the main strengths of the paper:
- Self-play approach for optimization modeling which does not rely on external data. Having worked in this space, training data for optimization modeling is indeed a huge issue. I generally like the idea of fine-tuning a model to generate high-quality data.
- The dual-role learning architecture. The proposer and solver are not trained as unrelated components, but rather two roles of the same shared-parameter model, trained alternately with role-specific prompts and rewards. Additional elegant features of the architecture include: using solver performance to filter and categorize the training instances and utilizing the reference buffer for subsequent training.
Critical weaknesses
- Methodological issues with the reward structure. While the system does assess solver-relative difficulty, difficulty is not part of the immediate proposer GRPO reward structure. On a related note, it's not clear to me how the system rewards for the diversity of the training examples. More importantly, as far as I can tell, the reward does not directly verify that the natural language description and declared formulation represent the same problem. (there are other checks such as canonical verification and code verification, but they do not guarantee equivalence).
- The empirical results are not particularly strong. While OPT-ZERO substantially improves its base model, it does not outperform the strongest specialized baseline (StepORLM). Critically, the evaluation also omits relevant small models such as Qwen3.5-9B and GPT-OSS-20B, which I believe should do quite good on the benchmarks. On a related note, performance on the harder benchmarks ( (OptMATH-Bench, MIPLIB-NL) is still fairly weak.
** Update after rebuttal: The authors have clarified how they framework ensures that the instances are diverse and complex enough (addressing weakness 1 above). They have also added experiments that show the merits of their framework on more powerful small models (addressing weakness 2 above). I am thus happy to raise my score to weak accept.
Questions for authors
- Please elaborate in detail how semantic consistency between the natural-language description and the declared formulation is verified. Even if the verification is currently partial, please suggest how it can be strengthened.
- How does the system ensure that the training set is diverse enough, e.g., covers a large amount of problem classes?
- Would the authors be able to compare against stronger SLMs such as Qwen3.5-9B and GPT-OSS-20B? As a stretch goal (perhaps for the future), it'll be great to see whether fine-tuned versions of these models could improve with the data generated by OPT-ZERO.
5 comments
Reply to Reviewer aL5J (2/3)
[W1.2, Q1] Semantic consistency between problem descriptions and formulations.
Thank you for raising this question. We clarify that our current verification pipeline combines three complementary mechanisms:
Generation requirements: The Proposer is instructed to faithfully express the declared formulation in the story, including all coefficients, constants, constraints, and variable bounds, without hidden parameters. This explicitly requires the LLM to generate consistent content.
Formal verification: The canonical verifier first checks whether the declared formulation is feasible and bounded, and solves it to obtain a reference optimum. The Proposer's code is then independently executed and checked against this value. These checks establish a reliable reference for subsequent story-based reconstruction, which helps assess whether the natural-language description can lead to a solution consistent with the declared formulation.
Story-only reconstruction: Before buffer admission, the Solver makes eight attempts using only the natural-language description. We retain problems with both objective-matching and non-matching outcomes, requiring at least one successful reconstruction from the story. During training, the reconstructed formulations and code are also evaluated against the reference objective. This helps ensure the consistency between objective and model generation.
These mechanisms provide practical consistency checks without requiring external supervision. However, objective agreement does not guarantee full semantic equivalence. We deliberately rely on automatically verifiable feedback rather than introducing a teacher model or human annotations into the training loop. In this way, we avoid hacking consistency by checking only the objective agreement.
To further assess semantic reliability, we conducted an human-expert audit of 50 retained training instances during Qwen3.5-9B self-play. We found that 74% exhibited full story–formulation consistency, while another 18% contained differences in the feasible region but preserved the correct optimal objective value. The remaining 8% involved changes to optimal solutions or incomplete descriptions. Thus, most of the audited instances either maintained full semantic consistency or preserved the correct objective target used for reward computation. These findings provide additional evidence for the reliability of the objective-based supervision while identifying remaining semantic discrepancies.
Further improvements. Recent research offers several promising directions for strengthening semantic verification. First, an independent LLM-based auditor could assess story–formulation consistency at the level of variables, objectives, and individual constraints, following approaches such as OptArgus [1] and Opt-Verifier [2]. Second, semantic alignment and correction mechanisms, such as SAC-Opt [3] and SemOPT [4], could provide more fine-grained feedback to detect and correct inconsistencies that objective matching alone cannot identify. While it is not in the scope of our current work, exploring these approaches as additional verification or reward mechanisms is a promising direction for future work that combines with self-play training. We have put this as a future work in Conclusion to highlight it as an effective way to enhance the self-play framework.
[1] Li et al. (2026). OptArgus: A Multi-Agent System to Detect Hallucinations in LLM-based Optimization Modeling. arXiv:2605.11738.
[2] Liu et al. (2026). Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification. ICML 2026.
[3] Zhang et al. (2026). SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modeling. ICML 2026.
[4] Zhou et al. (2026). SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search. Findings of EMNLP 2026.
Reply to Reviewer aL5J (3/3)
[W2, Q3] Model performance and training of more SLMs.
Regarding model performance: We believe the empirical contribution of our method lies in achieving competitive performance without externally curated optimization training data, together with improvements on harder external tasks. In summary, we already achieved:
1. State-of-the-art accuracy on individual benchmarks and in the data-augmented setting, achieving the highest accuracy among compared fine-tuned models on IndustryOR and ReSocratic (two harder benchmarks), as well as the highest Micro-average accuracy (91.3%) over all benchmarks with OPT-Zero-D.
2. Stronger zero-shot generalization across unseen solver interfaces (CPLEX, SCIP, and Pyomo), consistently outperforming specialized baselines and the initial backbone across all evaluated benchmarks.
3. Greater independence from external training data, achieving competitive performance through self-generated problems and solver-grounded feedback, without human-annotated training instances or teacher-generated solutions.
Comparison with StepORLM. OPT-Zero-8B achieves 90.3% Micro accuracy versus 90.4% for StepORLM. However, this comparison also involves substantially different training-data requirements: StepORLM uses 50K verified warm-up training instances, whereas OPT-Zero generates its optimization training problems through self-play,without an external task corpus or teacher-generated solutions. Reaching this level of performance demonstrates that self-generated supervision can support a competitive specialist model. Additionally, we want to highlight that when external data are incorporated into 30% of Solver training samples, the separate OPT-Zero-D variant achieves 91.3% Micro accuracy, significantly exceeding StepORLM. In this case, we saved 50K*70% instances and their heavy preprocessing work but achieved the strongest results.
Performance on harder benchmarks. OPT-Zero improves Gemma4-31B-IT from 41.0% to 48.2% on OptMATH-Bench and from 8.92% to 11.74% on MIPLIB-NL (Table 12). MIPLIB-NL derives from industrial-scale MILPs in MIPLIB and requires translating complex model structures and constraint relationships into executable formulations. The low absolute accuracy highlights a remaining gap between the model’s current capabilities and reliable industrial-scale optimization modeling. We observe a fact that most models cannot gain high accuracy rates in these two benchmarks, implying the difficulty of them for current LLMs.
Nevertheless, our improvements on both benchmarks show that self-play advances the capabilities of LLMs more significantly, compared to all the other fine-tuned baselines.
Training of more SLMs. We first evaluate the performance of GPT-OSS-20B and Qwen3.5-9B, and the results show that our trained 8B model (90.3 Micro Avg.) consistently outperforms GPT-OSS-20B (86.9) and Qwen3.5-9B (52.7). Following the suggestion of the reviewer, we also fine-tuned GPT-OSS-20B and Qwen3.5-9B by OPT-Zero, with results summarized below (and also included in Table 4 of the revised paper).
| Model / Setting | NL4Opt | EasyLP | ComplexLP | NLP4LP | IndOR | ReSocratic | Macro Avg. | Micro Avg. |
|---|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | 34.7 | 70.3 | 36.9 | 36.0 | 50.0 | 50.6 | 46.4 | 52.7 |
| + OPT-Zero | 93.0 | 89.7 | 64.9 | 96.1 | 57.1 | 91.1 | 82.0 | 88.5 |
| Absolute gain | +58.2 | +19.4 | +27.9 | +60.1 | +7.1 | +40.4 | +35.5 | +35.8 |
| GPT-OSS-20B | 91.1 | 93.0 | 55.9 | 91.0 | 52.4 | 86.6 | 78.3 | 86.9 |
| + OPT-Zero | 94.4 | 96.0 | 62.2 | 100.0 | 59.5 | 94.8 | 84.5 | 92.4 |
| Absolute gain | +3.3 | +2.9 | +6.3 | +9.0 | +7.1 | +8.2 | +6.1 | +5.5 |
These new results again verify that our self-play training method can generalize to more SLMs, although they have different initialization, training, and model sizes.
Reply to Reviewer aL5J (1/3)
Dear Reviewer aL5J,
We thank the reviewer for acknowledging the motivation of the study and the dual-role learning architecture of the self-play methodology. We address the raised concerns in a point-to-point way below.
[W1.1, Q2] Difficulty and diversity of the training set.
Thank you for raising these questions. Our framework incorporates difficulty adaptation and diversity control at complementary stages of the training loop.
Difficulty influences both Proposer learning and curriculum selection. Although the empirical solve rate is not used directly as the immediate Proposer reward, it determines the adaptive targets of the structural reward. Specifically, the targets for variable count, constraint count, and coupling density are updated between iterations using problems of medium Solver-relative difficulty (Section 3.2 and Appendix C), making the GRPO reward space adapted by the difficulty. Subsequent Proposer updates therefore receive structural feedback calibrated to the Solver’s current ability. Furthermore, Solve-rate estimates also guide task admission and difficulty-aware sampling (Section 3.4). This design incorporates Solver feedback without requiring multiple additional Solver rollouts for every completion during Proposer GRPO.
Our experiments support these design choices. Replacing the structured Proposer reward with a binary reward reduces Micro accuracy from 90.3% to 86.9% (Table 3). Moreover, Figure 7 shows that prioritizing harder problems consistently outperforms easy-first and medium-first sampling across all six benchmarks, with particularly noticeable gains on ComplexLP and IndOR.
Diversity is promoted through generation and buffer management, without a separate diversity reward term. Three mechanisms contribute:
Reference-conditioned generation: The Proposer prompt explicitly requests objectives and constraints that differ from the sampled reference model, together with a different application context for the story (Appendix B). Sampling different references encourages variation in both formulation and context.
Buffer-level diversity control: The reference buffer caps the number of problems sharing the same structural signature, comprising the LP/IP/MILP family, variable count, constraint count, and optimization sense (Section 3.4). Therefore, the buffer also prohibits the existence of many problems with similar structures.
Replay: We draw 20% of Solver training samples from the buffer, maintaining exposure to previously generated problems as new tasks enter the curriculum. Since the generated problems are random, the new tasks additionally inject diversity.
Appendix F.13 provides quantitative evidence for the resulting textual diversity. Among 300 retained buffer problems sampled across iterations, the Vendi Score is 244 and nearest-neighbor redundancy is 0.32, compared with 178 and 0.56 for 300 OptMATH-Train problems under the same TF-IDF analysis (Table 16). These results show greater textual diversity and less repetitive wording in the sampled training problems.