- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper considers the task of translating NL descriptions of nonlinear optimization problems into executable solver code. The proposed framework first extracts the optimization model using global and sentence-level passes (a step termed dual-granularity extraction) and then applies the NED-tree. NED-tree converts complex non-linear expressions into a so-called 'linear backbone', including auxiliary variable definitions and atomic operations; the underlying idea here is that this decomposition can be directly mapped to solver APIs. Finally, a subsequent coding and repair step produces the executable code. To evaluate the system, the paper introduces the NEXTOR benchmark, and shows sizeable accuracy improvements across multiple benchmarks.
Critical strengths
Thank you for submitting your work to ICLR. The main strength of the paper is in designing an end-to-end solution for the task of NL2Code for non-linear mathematical programs. This problem is more complex than linear programming as the translation to code is less direct; furthermore, the interpretation of the description is inherently more involved. On a related note, there is significantly less training data for non-linear problems, which may also explain the inferior results of existing language models.
Critical weaknesses
- Solution robustness (and current limitations). Section 5.3-5.4 report a clear advantage of using NED-Tree. That said, I would have liked to have more insights into the framework’s behavior when decomposition is difficult or unsupported. In particular, the paper can be made stronger with a more systematic analysis of unsupported cases, decomposition failures, and the effect on performance as expression complexity increases.
- Flexibility and adaptability of the system. The experiments appear to center on a particular set of operators and solver interfaces. Because optimization solvers continue to expand their native non-linear support (e.g., Gurobi 13, and the upcoming Gurobi 14), I would have liked to see explicit accounting for the 'stability' of NED-Tree - e.g., can it remain as is or does it have to evolve with a new set of solver APIs?
Aside from the above, I found the paper's presentation very hard to follow and below the standard of a top conference like ICLR. The authors used complicated terms and notations that are not intuitive. Furthermore there are no examples until fairly late in the paper. I would suggest the completely revise the paper, and introduce a running-example early on, which can be used to exemplify the different stages of the pipeline.
Questions for authors
- Following my comment from above, Sec. 5.3 shows a substantial ~26 percentage-point accuracy drop when NED-Tree is removed, and Sec. 5.4 reports successful preservation and mapping results on six tested expression patterns. At the same time, Algorithm 1 can return UNSUPPORTED, and the conditional equivalence result in Appendix I excludes unsupported operators, cyclic or incomplete definitions, and missing domain enforcement. Can the authors please report the frequently of such cases and provide a taxonomy of unsupported or failed decompositions. If possible, please show how performance varies with expression depth, number of operators, etc.
- Can the authors suggest potential future approaches to overcome failures as outlined above?
- It seems to me that the prompting-based techniques can be carried over in order to fine-tune models. Can the authors comment on that, and do they anticipate that those guidance techniques will eventually be absorbed in model training?
12 comments
Response to Reviewer ch2r(2/7)
W1 Q1 Q2: Unsupported or failed decompositions and potential improvements
We analyzed 255 questions individually to study failure frequency, the failure types observed, and the effect of expression complexity. We also compared code generation using fixed mathematical inputs and ran an independent test of first code-generation attempts on pure mathematical models. NEXTOR's nonlinear subset contains 38 questions. We also included 183 questions from OptiBench and 34 from OptMATH.
Table 1. Datasets and Evaluation Scope
| Dataset / set | Number of questions | Composition and purpose |
|---|---|---|
| NEXTOR nonlinear set | 38 | More complex nonlinear tasks; end-to-end testing and paired comparison at the coding stage |
| OptiBench nonlinear set | 183 | External task set; comparison with fixed inputs |
| OptMATH nonlinear set | 34 | External task set; comparison with fixed inputs |
(a) Bad-case statistics and analysis
First, we analyzed Auxiliary-stage failure records for GPT-6-sol (medium) on 255 questions. The table classifies the outputs and validation results in the logs. Empty candidates, validation rejections, service errors, and cases with no mathematical input from the upstream stage are listed separately.
Table 2. Auxiliary-Stage Outcomes on 255 Questions
| Failure or anomaly category | Number / all 255 questions | Details and handling |
|---|---|---|
| Nonempty candidate tree passing structural checks | 149(58.43%) | Construction information proceeds to subsequent coding; passing structural checks does not mean mathematical equivalence has been proven |
| Inconsistent symbol binding or auxiliary-variable naming | 32(12.55%) | 13 cases where symbols/indexed elements were unbound; 19 cases of auxiliary-variable name conflicts across candidates |
| Beyond current expression/subscript parsing support | 11(4.31%) | 8 cases of unsupported subscript forms; 3 cases where expressions or functions failed parsing |
| Returned field structure did not meet the contract | 3(1.18%) | Output did not meet the field requirements of the construction interface |
| Decomposition did not correspond to the original expression/auxiliary definitions | 3(1.18%) | 2 cases did not correspond to the source expression; 1 linearized form did not reference a declared auxiliary definition |
| No decomposition candidate returned | 42(16.47%) | Empty candidate, recorded separately and not directly classified as a mathematical error |
| Auxiliary request failed | 2(0.78%) | API connection failure |
| No mathematical input generated upstream | 13(5.10%) | Did not enter the Auxiliary stage |
Among the 242 questions that reached Auxiliary, 49 failed the structural/interface checks (20.25%). Of these, 32 involved symbols or naming, 11 involved current parsing support, and 6 involved output fields or expression correspondence. The implementation rejects these candidates and falls back to the original mathematical model. Including 2 service errors, there were 51 fallback records, and 42 were ultimately scored as correct. This shows that the workflow can still continue solving with the original mathematical model when decomposition checks fail, providing some robustness.
The 42 questions with no returned decomposition candidate are recorded separately as no_decomposition. Coding uses the original mathematical model and any returned modeling suggestions, without a decomposition candidate. These 42 questions are not included in the 51 fallback records, which consist of 49 validation rejections and 2 service errors. All 42 empty-candidate cases produced answers that could be scored; 30 were correct (30/42, 71.43%).
Response to Reviewer ch2r(3/7)
(b) Further checks of the four decomposition issues you identified
To directly address your questions about unsupported operators, cyclic dependencies, incomplete definitions, and domain enforcement, we further reviewed the trajectories of 255 questions. We checked the original candidates and their generated code, rather than classifying cases only by interface errors.
Table 3. Decomposition Checks and Observed Frequencies
| Item checked | Frequency | Checks and specific findings |
|---|---|---|
| Candidates containing operators unsupported by the current constructor | 1/198 (0.51%) | OptiBench:345 directly uses in an auxiliary definition, outside the function set supported by the current candidate parser. This is an implementation support limit; it does not mean that the problem cannot be solved. |
| Cyclic dependencies in auxiliary definitions | 0/198 (0.00%) | We rebuilt the dependency graph from the actual references on the right-hand sides of definitions and checked for self-references and cycles across candidates. Repeated use of the same auxiliary name was handled separately, rather than directly counted as a cycle. |
| Incomplete auxiliary definitions | 6/198 (3.03%) | Candidates referenced quantities that were neither original variables/parameters nor defined in the auxiliary definitions. For example, OptiBench:70 defines the numerator and denominator , but not the objective ; NEXTOR:37 lacks a definition of the cost quantity . |
| Necessary domain conditions omitted during decomposition | 1/66 (1.52%) | 66 questions involve variable denominators, roots, logarithms, or non-integer powers. NEXTOR:2 does not retain the nonzero condition when eliminating the denominator, allowing a zero at which the original fraction is undefined. This accounts for 0.51% of all 198 questions with candidates. |
| Necessary domain safeguards not enforced in the final code | 0/43 (0.00%) | Among these 66 questions, 43 passed the checks and proceeded to tree-guided coding. We checked variable bounds, associated constraints, and operator implementations in the final retained code and found no such omissions. |
The table shows that the issues you raised occur relatively rarely. The main issue is incomplete auxiliary definitions, which may stem from the modeling stage, while the NED-Tree component has almost no bad cases.
(c) Potential future methods for the main failure issues
Based on the results in (a–b), we identified the main failure categories and the following possible approaches to address them:
Table 4. Potential Improvements by Observed Category
| Observed problem category | Potential improvement step |
|---|---|
| Symbol binding/auxiliary naming issues | Establish a shared symbol table and unique names; after renaming, check that each reference points to a unique, defined object |
| Expression/subscript parsing rejections | First normalize finite indices, then add parsing forms with explicit mathematical semantics and domain rules |
| Field or expression correspondence issues | Check the correspondence among output fields, source expressions, and candidate definitions; validate along the dependency graph that every intermediate quantity is defined and the graph is acyclic |
| Domain enforcement and generated-code interface errors | Attach domain conditions to local operations and check that the code enforces them; check function names, input/result-variable order, and outer relations for each backend |
| Incomplete auxiliary definitions | Use more detailed modeling, establish a traceable modeling process, and try multiple backbones |
Response to Reviewer ch2r(1/7)
Dear Reviewer ch2r,
Thank you for your thoughtful review. We appreciate your recognition of the value of an end-to-end approach to nonlinear optimization modeling. Below, we provide additional explanations and experiments addressing your comments. We have also updated the paper PDF. If you have any further questions, we would be happy to discuss and address them.