- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper under review considers LLM-based optimization modeling. The authors demonstrate that LLMs do not faithfully output optimization formulations. In particular, they show that models might omit requirements and/or associate values with wrong entries: the code they generate may still be executable and produce an optimal solution, but to not to the correct problem. To address these issues, the paper designs and implements TAGO. The idea behind TAGO is to train models to express an optimization problem through five ‘evidence-annotated’ components: sets, parameters, variables, objective and constraints. The training pipeline relies on a new synthetic training dataset, termed StepTrace-OR, where each optimization problem includes a mathematical formulation whose individual modeling steps are linked to supporting quotations from the problem statement, together with executable Pyomo code. The pipeline then applies evidence-aligned supervised fine tuning followed by RL. The RL rewards capture evidence-expression consistency, evidence coverage, formatting and solver outcomes. Experiments on five optimization modelling benchmarks report improvements over the same backbone baseline models.
Critical strengths
-
The paper zooms-in into the shortcomings of LLMs for modeling optimization. I liked the breakdown analysis, distinguishing between mapping misalignment and information omission.
-
The paper designs a full training pipeline to address the identified issues, including the stepTrace-OR trace, mutistage filtering, careful reward design and an elegant combination of SFT and post-training RL. The results in Table 1 show improvements on multiple backbone models (compared to other SLMs derived from these models), and ablation study (Table 2, Sec 4.3) confirms that all components of the solution contribute.
Critical weaknesses
While the paper is fairly well written, I unfortunately see a major issue with its relevance.
-
Main weakness: Unclear value compared to stronger models. TAGO teaches relatively small Qwen models to decompose the problem, ground each modeling step in source text, and verify coverage. However, the experiments do not really establish that this produces a new capability. Stronger SLMs and frontier reasoning models may already possess much of the underlying source to formulation alignment capabilities. Even if they do not expose this behavior by default, it may be elicited through a structured prompt requiring, e.g., the same five components, supporting quotations, and a final omission check. As far as I can tell, even the comparison with GPT-4o-based agentic systems doesn't use a matched "TAGO-style" prompt and/or verification protocol. Consequently, the reader is left unconvinced whether specialized traceability training is required with the fast evolvement of reasoning LLMs.
-
Incomplete formulation-level evaluation and unclear benchmark versions. The main Pass@k metric in Sec. 4.1 considers a completion correct if it is feasible and matches the reference objective, although an incorrect formulation may produce the correct objective. Sections 4.4.1 and 4.4.2 provide useful evidence and error analyses, but they do not measure how frequently this occurs among successful outputs (I am aware it's a hard problem!). More importantly, the paper does not seem to specify the exact versions of IndustryOR, MAMO, NL4OPT, and NLP4LP used in the experiments. The reviewer is aware that the original releases contained substantial errors, including incorrect formulations and reference objectives, and several cleaned versions now exist with different corrections, exclusions, and instance counts. This is particularly relevant (and important) for this paper: erroneous benchmark annotations can affect not only objective-based accuracy, but also the correctness of the five formulation components, the validity and completeness of their evidence links, and the classification of omissions and misalignments. Without verified problem statements, formulations, and reference objectives, it is difficult to determine whether the reported traceability improvements reflect better modeling.
Questions for authors
In relation to weakness 1 above,
-
Can the authors check how does TAGO compare with stronger open-weight models, such as Qwen3.5-9B and GPT-OSS-20B, under the same evaluation protocol? Evaluating these models without additional training would help determine whether TAGO’s gains persist when compared with more capable base models.
-
Have the authors evaluated frontier LLMs using both a standard optimization-modeling prompt and a 'TAGO-like prompt' (e.g., explicitly requests the five formulation components, supporting evidence for each component, and a final completeness check?
In relation to Weakness 2 above,
- Which exact versions of the five evaluation benchmarks were used, and how were their problem statements, formulations, and reference objectives verified?
Dear Reviewer,
The rebuttal period is nearing its end. We kindly ask you to read the authors' response, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.
Best regards, Area Chair