- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ASTOR, a grammar-guided framework for translating natural-language optimization problems into executable formulations. It also introduces SoftOR, a 355-instance benchmark for weighted and lexicographic soft constraints. ASTOR achieves 88.5% accuracy, outperforming the strongest baseline at 84.8%.
Critical strengths
- SoftOR addresses an important gap in evaluating soft-constraint optimization.
- ASTOR provides a clear, training-free framework with promising improvements, particularly on TravelPlanner.
Critical weaknesses
- Ambiguity in weighted evaluation. Qualitative preference rankings do not uniquely determine numerical weights or optimal solutions. Evaluating against hidden original weights may penalize valid formulations.
- Limited ablation coverage and generalizability. The ablations cover only 45 soft-constraint instances, including just two weighted-penalty cases. ASTOR does not consistently outperform Codegen across backbones. Moreover, Gemini 3.8 Flash is evaluated only on Nurse Rostering, and newer GPT models are not included.
Questions for authors
- If different numerical weights preserve the same qualitative preference ordering but yield different optimal solutions, how can the hidden-weight evaluation justify a unique gold answer?
- Given the relatively modest benchmark size and reported API costs, could the authors extend the key ablations to the full 355-instance SoftOR benchmark and conduct full-benchmark comparisons using Gemini 3.8 Flash and newer GPT models?
2 comments
Thank you for the detailed response and additional experiments. After careful consideration, I have decided to maintain my original rating.
1. Investigating and ablating weights on Nurse Rostering Our reason for withholding the numerical weights was to study what happens when a natural-language problem describes preferences without specifying their exact penalties. In that setting, the model has to infer the weights as part of formulating the problem. We wanted to evaluate this additional challenge as a part of the modeling task. In fact, we had felt that giving the weights could be potentially too advantageous to the more sophisticated methods, so we withheld them. However, we actually ran experiments both with and without explicitly given weights all along, and we are happy to now report both.
Here in this revision, we now also report the complementary with-weights experiment: we explicitly provided the numerical weights in the prompt for all 15 Nurse Rostering instances for each route. With the weights given, ASTOR achieves the highest observed accuracy, solving 14 of 15 instances (93.3%), followed by CoE at 13 of 15 (86.7%). The full results are below.
These results can also be found in Table 13 within Section A.4 of the appendix.
2-a. Extending the ablations. Thank you for this suggestion. In this revision, we have now conducted all the ablations on all 355 instances of SoftOR, which includes testing other baseline methods and similar variantsto isolate the gains produced by ASTOR, as now reported in Figures 5 and 6 and Tables 6 and 7. To address the JSON format, we include OptiMUS 0.3 as a baseline in our main experiment, which also uses a structured JSON as an intermediate optimization-model representation. ASTOR achieves more than double the performance of this baseline. To address the deterministic compiler component, we include an ablation setting that uses an LLM (Gemini 3.8 Flash, the same as our new main experiment) to compile from JSON to gurobipy, and we see that this reduces performance by 4.8 percentage points. To isolate the contribution of explicitly constructing a formal AST from that of grammar-based guidance alone, we include a grammar-guided code-generation ablation, which reduces performance by 7.6 percentage points. This addition complements our prior ablation of the grammar-guided direct-solve baseline, which reduces performance by 20.0 percentage points. We also include ablations on the temperature setting of the LLM backbone to test ASTOR’s stability under varied outputs, and we see that its accuracy remains stable. The ablations suggest ASTOR’s gain comes from the JSON-format of the formal grammar and deterministic parsing. In Figure 11d we can see that the human readable guidance mostly saves token cost. See Figures 5 and 6 and Tables 6 and 7.
2-b. Backbone coverage and generalizability. In this revision, we also reran all experiments so that they are evaluated in a setting that is both unified and equipped with the latest capabilities All of our main experiments here now use the new LLM backbone Gemini 3.8 Flash. In this run, we see that ASTOR’s accuracy is statistically significantly higher than all other routes. ASTOR leads (88.5%) compared to Codegen (84.2%) ORThought (85.4%), CoE (85.6%), CoT (83.1%) , Direct (63.9%), and OptiMUS (41.7%). To address the need for modern multi-agent routes, we also ran SAC-Opt (ICML 2026), a multi-agent method on all 355 instances of SoftOR using Gemini 3.8 Flash. We observed that about 38 of its responses exceeded Gemini’s max token limit, but on the instances that it did produce an answer, it had an accuracy of 21.5%. We also reran the experiment on ASTOR and its top competitors using GPT 6 as the LLM backbone, and we see that under this setting ASTOR is statistically significantly better than all methods except for Codegen, which it still outperforms. ASTOR leads (87.6%) compared to Codegen (86.5%), ORThought (84.5%), CoE (84.5%), and CoT (83.9%).
In total, for this revision we newly ran all 355 instances of SoftOR through eight routes for the main experiment on Gemini 3.8 Flash, 11 different ASTOR ablation settings, and five routes for the new LLM backbone ablation on GPT-6. Please see Tables 3,6,7 and Figures 5,6.