acceptodds
Under review as a conference paper at ICLR 2027

Semantic Exploration for Underspecification in LLM-Based Optimization Formulation

Abstract

Large language models (LLMs) can translate natural-language optimization problems into mathematical formulations. However, natural-language descriptions are often underspecified or ambiguous, and successful solver execution can leave these issues undetected: a solvable formulation need not capture the intended problem. We introduce DIVERSEFORM, a formulation assistant that explores alternative mathematical interpretations through history-conditioned diverse sampling. Each sampling step includes previous formulations and requests a different defensible interpretation. We design a declarative intermediate representation that simplifies modeling by directly expressing mathematical variables, constraints, and objectives. This representation makes differences between candidate formulations explicit, facilitating comparison during the search. Experiments show that DIVERSEFORM outperforms existing prompting-based approaches using frontier models and surpasses a state-of-the-art open model fine-tuned specifically for optimization modeling, without requiring additional training. Further analysis shows that semantic exploration helps identify distinct interpretations and uncover underspecification that calls for clarification.

Primary area:
optimization
Keywords:
Auto-formulation, Optimization Modeling, Large Language Models, Ambiguity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_s8FROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DiverseForm, a training‑free formulation assistant that translates natural‑language optimization problems into mathematical formulations under underspecification. DiverseForm employs history‑conditioned diverse sampling: each subsequent query incorporates earlier encodings along with their solver statuses, prompting the model to propose a different defensible interpretation of the problem description. A declarative intermediate representation (IR) is adopted to explicitly capture variables, constraints, and objectives, thereby surfacing differences among candidate formulations. Evaluated on four optimization‑modeling benchmarks (NL4Opt, MAMO ComplexLP, IndustryOR, and ComplexOR) using Sonnet 4.6 and Gurobi, DiverseForm achieves 97.4% overall Pass@5, outperforming all prompting‑based and agentic baselines. In the open‑model study, Qwen3‑14B with thinking enabled surpasses the task‑specifically fine‑tuned LLMOPT in overall accuracy without additional training. Further analysis covers ambiguity‑injection recovery, lexical versus numerical diversity, and case studies of interpretation categories.

Critical strengths

This work identifies a practically important and under-explored failure mode of LLM-based auto-formulation: executable but wrong formulations caused by underspecified natural language, which escape execution-based debugging. The proposed history-conditioned exploration yields substantial coverage gains (from 89.2% to 97.4% overall Pass@5 with Sonnet 4.6), outperforming the strongest baseline by 5.1 percentage points with fewer than half as many model calls (5.0 versus 11.5). The accompanying diagnostic studies (interpretation categories, ambiguity injection, lexical-versus-numerical diversity) provide useful insights into where problem descriptions require clarification, which may inform future research on clarification-oriented formulation assistants.

Critical weaknesses
  1. The paper exhibits limited conceptual novelty and largely resembles a solid prompt-engineering study tailored for the auto-formulation setting. The core mechanism, conditioning each new sample on previous generations and instructing the model to take a different defensible interpretation, is a direct application of well-known diversity-promotion ideas from recent literature (repeated sampling in self-consistency, conditioning on previous responses in integrative decoding, and guided diversity in G2). The exploration is also heavily scaffolded: the sampling prompt in Appendix B.2 itself enumerates six kinds of interpretation decisions for the model to take differently, so candidates largely vary along pre-specified dimensions rather than discovered ones. The declarative IR, the second pillar, is closely related to the meaning representations of NL4Opt and the Canonical IR, and its own ablation shows a modest 0.9-point overall gain for the frontier model, with executable code even surpassing the IR on IndustryOR (95.7 versus 93.8); the benefit is concentrated in the smaller model (+4.4 points for Qwen3-8B). Notably, one-shot enumeration with an interpretation guide matches DiverseForm’s overall coverage (97.4%) in a single call (Table 4). It remains hard to disentangle how much of the improvement comes from the proposed iterative exploration paradigm versus simply drawing more samples with an explicit diversity instruction.

  2. The experimental evaluation has several notable drawbacks. (1) The headline metric is oracle Pass@5 under a 5% numerical tolerance. Numerical agreement does not establish formulation correctness (as the paper itself acknowledges), and no candidate-selection mechanism is proposed, so the reported coverage overstates what a deployable formulation assistant could deliver. The metric also conflates resolving genuine underspecification with guessing the annotator’s reading: the paper’s own case studies show that references impose a stated preference in one problem but omit it in another (NL4Opt 143 versus 202). In addition, all headline numbers are computed on an adjudicated 495-instance subset after excluding 79 instances, whose documentation is itself inconsistent: Table 6, captioned as complete, lists only 71 exclusion identifiers (e.g., 16 of the 22 NL4Opt exclusions, and no IndustryOR entry despite one exclusion in Table 5). (2) The headline comparison runs each baseline at its own call budget (1.7 to 31.4 calls per problem), so it conflates the exploration strategy with workflow design. The paper’s budget-controlled comparisons tell a weaker story than the headline 5.1-point margin: under the same five-proposal budget, independent sampling trails history-conditioned sampling by only 2.8-4.9 points (Figure 4), and guided one-shot enumeration matches the overall coverage in a single call, though that call may emit an uncapped number of candidates evaluated under Pass@any, and its interpretation guide was derived from the recovered cases, i.e., evaluated in-sample (Appendix B.4), with no held-out validation reported. (3) The ambiguity-injection study is small and constructed: 59 variants are derived from 56 problems of the elementary NL4Opt benchmark, the construction only retains variants whose alternative reading changes the optimum (a selection bias toward numerically observable ambiguity), only one model is tested, and only recovered cases are analyzed, with no inspection of the problems exploration still misses. The reported numbers are also internally inconsistent: Section 4.5 reports 59 variants with 16 independent-sampling misses and 15 recoveries, whereas Appendix E.1 describes 26 edited descriptions with 8 misses, all recovered, and the appendix further admits that raw traces were unavailable in its snapshot, leaving the recovery counts not trace-verifiable. (4) No variance or statistical significance is reported for any comparison despite sampling at temperature 0.7. The margins are sometimes razor-thin: on ComplexOR, with only 17 retained instances, the reported advantage over Standard and OptiTree (88.2% versus 82.4%) corresponds to exactly one problem, so it is unclear whether the claimed per-benchmark wins are stable across runs.

Questions for authors
  1. Figure 4 compares independent sampling with history-conditioned diverse sampling, but the two conditions differ in both the displayed history and the explicit diversity instruction. Can you isolate the contribution of history conditioning itself (e.g., history without the diversity instruction, or the instruction without history)? Relatedly, since the interpretation guide was derived from the recovered cases (in-sample, per Appendix B.4), how does guided one-shot enumeration perform on held-out problems or unseen interpretation categories?

  2. DiverseForm leaves candidate selection to human review, yet all headline results are reported as oracle Pass@5. How should the final formulation be selected in practice, and what is the end-to-end accuracy under a realistic non-oracle selection mechanism? Relatedly, could you reconcile the discrepancy between Section 4.5 (59 variants, 16 independent-sampling misses, 15 recoveries) and Appendix E.1 (26 edited descriptions, 8 misses, all recovered)?

2 comments

Q1. Novelty beyond existing diversity prompting and intermediate representations.

A1. We agree that neither structured representations nor history-conditioned generation is new on its own. Our contribution lies in how we combine these components to generate and explore different mathematical interpretations of the same natural-language(NL) problem. Together, these findings show how declarative generation and diverse sampling can improve accuracy while making unresolved assumptions visible for human review.

Building on the structured representation established by NL4Opt and Canonical IR, we contribute a concrete generation procedure using declarative syntax in the prompt, enabling the LLM to encode mathematical formulations directly from the NL description. Our syntax-based prompting significantly reduced failures due to execution-based errors on both large and small models compared to the other baselines.

Sonnet 4.6 (Figure 2)

MethodNo program emittedRuntime/transpilation error
AF-MCTS511
DiverseForm01

Qwen3-8B with thinking disabled (Table 8)

MethodExecution/output failure rate
AF-MCTS69.0%
DiverseForm27.8%

Combined with the declarative representation, our method promotes exploration in the structured semantic space, making resulting modeling choices explicit and comparable, unlike G2, which promotes diversity through token-level decoding interventions informed by prior outputs. The code-versus-IR ablation indicates that the representation complements this exploration.

Q2. Limits of oracle Pass@5 and candidate selection without reference answers.

A2. DiverseForm is intended as a formulation assistant that presents a small set of semantically valid formulations for human inspection. Our Pass@5 metric measures a narrower property: whether at least one candidate produces an interpretation that matches the intended problem.

However, we agree that reliable candidate selection remains important beyond the numerical agreement. As a supplementary analysis, we add the LLM defensibility evaluation. The result shows that an LLM applies conservative screening to flag defensibility, preserving 87.9% of reference-matching candidates for human review. Conversely, 4.2% of candidates judged defensible did not match the reference, potentially reflecting valid alternative interpretations or evaluator errors. We therefore explore this assessment as a potential aid to human inspection, with independent expert validation left to future work.

Q3. Separating history conditioning from explicit diversity instructions and predefined categories.

A3. We thank the reviewer for identifying the interpretation scaffolding in the original prompt. We removed the six predefined interpretation categories from Appendix B.2 and reran the main evaluation using the revised prompt. DiverseForm now achieves 98.5% overall reference-answer coverage with Sonnet 4.6, compared with the previously reported 97.4%.

For Figure 4, we rerun three frontier models with the revised prompt to evaluate the contribution of history conditioning compared with independent sampling. It improves coverage over independent sampling for all three frontier models. These results show that the gains do not depend on supplying the six predefined interpretation categories.

Q4. Guided enumeration as a competing baseline: candidate budgets and held-out generalization.

A4. This experiment shows why we chose successive generation of new candidates through iterative calling, adding prior outputs to the prompt. The revised results in Table 4 show that DiverseForm achieves 98.5%, compared with 95.2% for unguided enumeration and 97.4% for guided enumeration. Enumeration uses one model call to enumerate at most five encodings. The result also shows that history conditioning is better at finding possible interpretations, even without interpretation guidance, than case-derived interpretation guidance during enumeration.

Q5. Fair call-budget comparisons and statistical reliability of reported gains.

A5. We have clarified that the “Calls per candidate” column in Table 1 reports the number of LLM invocations required to produce one complete candidate. To assess sensitivity to stochastic generation, we repeat each sampling condition in Figure 4 three times for each model and report the mean Pass@5.

Q6. Scope, selection bias, and inconsistent reporting in the ambiguity-injection study.

A6. We acknowledge the study’s limited scope. We formed new ambiguity-injection cases across multiple benchmarks, verified to preserve the original reference optimum while introducing an alternative interpretation with a different optimum. Diverse sampling recovered all six cases that independent sampling missed, supporting its ability to explore interpretations overlooked by independent sampling. We reported these six cases in Appendix F and updated the results in Section 4.5 and Figure 5.

Dear Reviewer s8FR,

Thank you again for your constructive feedback and for taking the time to review our paper. As the discussion period nears its end, we wanted to follow up to see whether our responses, new experimental results, and paper revisions have addressed your main concerns. Please let us know if any points remain unclear or if further clarification would be helpful. We would be very glad to discuss further.

These are the new/revised experimental results:

  • Main result of Diverseform without any instruction guidance
  • MCTS-based search method as a baseline
  • LLM evaluation of candidates over diverse prompting
  • New problem set for ambiguity-injection experiment with non-trivial ambiguity cases

Best regards,

The Author

Reviewer_WpxGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies underspecification in natural-language optimization problems and proposes DIVERSEFORM, a training-free formulation assistant.DIVERSEFORM generates several formulations of an optimization problem by showing the LLM its previous formulations and solver statuses, then asking it to explore another defensible reading. It uses a declarative representation for variables, constraints, and objectives. The paper reports 97.4% numerical-answer Pass@5 with Sonnet 4.6 on four benchmarks. Comparisons with independent sampling, representation ablations, and an ambiguity-injection study examine the benefit of exploring alternative interpretations. The output is a candidate set for human review, rather than a selected final formulation.

Critical strengths
  1. The paper addresses a practical limitation of execution-based feedback: successful optimization does not show that a formulation captures the intended assumptions. Exposing alternative formulations and their modeling choices is a useful assistant design. Sections 3.1 and 4.6 also distinguish semantic alternatives from textual variation and numerical disagreement.
  2. Table 3 shows that exploration also works with executable code: Sonnet 4.6 reaches 96.5% Pass@5, versus 97.4% with declarative IR. The IR gain for Qwen3-8B-Thinking is 4.4 percentage points. This separates representation choices from exploration and identifies a larger representation benefit for the smaller model.
Critical weaknesses
  1. The provenance of the ambiguity-recovery evidence is unclear. Section 4.5 and Figure 5 report 59 variants from 56 problems, with 15 of 16 independent-sampling misses recovered. Appendix E.1 instead describes 26 variants and 26 controls, followed by eight misses recovered in each of two runs. It also says raw traces were unavailable in the snapshot used to compile the appendix. These may be different study versions, but their relationship is unexplained. This is a provenance and reproducibility concern for the ambiguity-recovery claim: its supporting records are not clearly identified. It does not establish that the result is wrong.
  2. The experiments support reference-answer coverage more directly than semantic validity. The wording intervention retains the original reference and selects alternatives whose objective values differ beyond the 5% threshold. Appendix E.2, for example, changes "at least 100" to "must use 100" while retaining the lower-bound target. Recovering this target measures numerical coverage, but does not establish that it is a defensible reading of the revised text. The authors acknowledge the limitation and describe case interpretations as provisionally plausible. A systematic independent assessment is still needed to distinguish genuine alternatives, equivalent rewrites, and modeling errors, and determine which explains the improvement.
Questions for authors
  1. Please reconcile the 59- and 26-variant accounts and identify the IDs and per-method outcomes supporting the 15/16 result. The 16 listed examples have the same category counts as the main-text misses; are they that subset? Does reference-based stopping only end evaluation after success, or can it influence generation?
  2. Could independent annotators, blinded to method and reference outcome, assess the recovered cases and a limited sample from both methods? Suggested categories are defensible distinct interpretation, equivalent reformulation, invalid formulation, and unresolved, with explicit-wording violations identified. Please report criteria, sample sizes, per-method proportions, agreement, and disagreement resolution. This would test the existing semantic claim.
  3. How many independent runs underlie Figure 4 and the main tables? For the reported 2.8–4.9 percentage-point gains over independent sampling, please provide paired uncertainty estimates and, if available, repeated five-proposal runs on a limited subset to assess stability.
  4. What limits and prompts produced Table 4? Section 4.3 describes enumeration without a cap, whereas B.3 sets k to five. Please give actual candidate counts and explain the case-derived guide in B.4. Its in-sample origin and Pass@any versus Pass@5 limit a strict matched-budget comparison.
2 comments

Q1. Independently assessing semantic validity.

A1. We agree that independently assessing semantic validity is needed beyond the numerical agreement. As a supplementary analysis, we add the LLM defensibility evaluation using GPT-6-Astra. The result shows that an LLM judge applies conservative screening to flag defensibility, preserving 87.9% of reference-matching candidates for human review. Conversely, 4.2% of candidates judged defensible did not match the reference, potentially reflecting valid alternative interpretations or evaluator errors. We therefore explore this assessment as a potential aid to human inspection, with independent expert validation left to future work.

We reassessed the 59 existing ambiguity-injection cases to determine whether their intended formulations represented defensible, meaningfully different interpretations. For assessment, we used LLM defensibility evaluation comparing the original problem and the edited problem. We retained 22 cases and added 14 newly generated cases, yielding 36 instances across multiple ORBench benchmarks rather than NL4Opt alone. Of these, one passed the strict qualification criteria, and 35 remained uncertain; The edits therefore preserve the original reference interpretation and its optimum while introducing an alternative interpretation with a different optimum.

In the ambiguity injection results, diverse sampling recovered all six cases that independent sampling missed, supporting its ability to explore interpretations overlooked by independent sampling.

Q2. Inconsistent reporting in the ambiguity-injection study.

A2. With new ambiguity-injection study, We report the updated results in Section 4.5 and Figure5 and document six recovered cases in the Appendix F.

Q3. Clarifying whether reference-based stopping influences candidate generation.

A3. Previously, we used reference-based stopping only in the ambiguity injection experiment for computational efficiency, as it is a supplementary study comparing reference-accuracy coverage between diverse sampling and independent sampling.

Q4. Reporting run counts, paired uncertainty, and stability of exploration gains.

A4. We agree that point estimates alone are insufficient to establish the stability of the reported gains. We reran three frontier models for each sampling method three times and reported the mean Pass@5 in Figure 4.

Q5. Clarifying enumeration prompts, candidate limits, and the comparability of Table 4.

A5. DiverseForm uses five iterative calls to generate a new candidate each time, and both enumeration experiments use one modal call to enumerate at most five encodings. DiverseForm and the unguided enumeration do not use any interpretation guide. Only guided enumeration uses the interpretation guide. Therefore, Table 4 shows that our method is better at finding possible interpretations, even without interpretation guidance, than case-derived interpretation guidance during enumeration.

Dear Reviewer WpxG,

Thank you again for your constructive feedback and for taking the time to review our paper. As the discussion period is nearing its end, we wanted to follow up to see whether our responses, new experimental results, and paper revisions have addressed your main concerns. Please let us know if any points remain unclear or if further clarification would be helpful. We would be very glad to discuss further.

These are the new/revised experimental results:

  • LLM evaluation of candidates over diverse prompting
  • New problem set for ambiguity-injection experiment with non-trivial ambiguity cases

Best regards,

The Authors

Reviewer_FkmEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors proposed DiverseForm, a framework to autoformulate optimization problems from natural-language descriptions. The key ideas are (1) using history of formulations and solver results and (2) encouraging the LLM to explore alternative modeling choices through prompting.

The authors demonstrated the improved accuracy over selected baselines, and performed various studies of the propose method (e.g., ablation, effect of budget, LLM used).

Critical strengths

Extensive Evaluations. The authors did various studies on the proposed method.

  • Ablation study showed that the two major components, namely history-dependent exploration (Figure 4) and diverse sampling (last two rows of Table 1), did play a large role in improving the performance.
  • A variety of LLMs, both open and closed, thinking and non-thinking, and different model families, were used to evaluate the proposed method.
Critical weaknesses

Novelty of Proposed Method. Neither of the two key ideas, namely using history and exploring diverse formulations, is new. This is particularly true for exploration, because most existing works in autoformulation ask the LLM to generate multiple candidates in various stages of the pipeline. I would argue that all the works employing MCTS, which are a lot of works, perform exploration. Examples of related works include:

  • PARM: Pipeline-Adapted Reward Model, https://arxiv.org/abs/2604.18327. multiple candidates are generated and subsequently scored by the respective reward models, with the highest-scoring outputs sampled as the final formulation and solution.
  • SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search, https://aclanthology.org/2026.findings-emnlp.185.pdf. applies a solver-informed evaluation; activates semantic-guided MCTS.
  • Dual-Cluster Memory Agent: Resolving Multi-Paradigm Ambiguity in Optimization Problem Solving. Central to this framework is the Dual-Cluster Construction, which resolves structural ambiguity by organizing historical data, [...], into a bipartite graph that [...].

While the authors did an extensive study on their own method, there is a lack of comparisons with these related works, either in experiments or even in discussions.

Questions for authors

Please see my main concern in Critical Weaknesses. Below are additional questions.

  • Could you give a clearer problem definition? On one hand, the motivation was "natural-language description is underspecified when it leaves a modeling choice unresolved and different defensible resolutions yield substantively different mathematical formulations" (Line 143). On the other hand, the methods are mainly evaluated based on accuracy of formulation. If the description was underspecified, multiple formulations can be correct. How do you judge whether a formulation is correct?
  • In Figure 4, the improvement by history-dependent exploration is highly dependent on benchmarks and LLMs used. Could you give some insights?
3 comments

Q1. How DiverseForm differs from other search-based literatures

A1. We thank the reviewer for identifying these relevant works. We have added AF-MCTS as a search-based formulation baseline. Our contribution has three aspects:

Direct declarative generation without intermediate modeling stages. We provide declarative syntax and encoding rules directly in the prompt, enabling each formulation to be generated without separate strategy, LaTeX, and code-generation stages. Figure 2 shows fewer execution-related failures for DiverseForm, supporting its practical reliability.

Sonnet 4.6 (Figure 2)

MethodNo program emittedRuntime/transpilation error
Standard141
Chain-of-Experts29
OptiMUS01
OptiTree40
AF-MCTS511
DiverseForm01

History-conditioned exploration of alternative interpretations. DiverseForm conditions each proposal on previous formulations and requests a different defensible interpretation. The declarative representation makes the resulting modeling choices explicit and comparable. In our main comparison, DiverseForm achieves higher reference-answer coverage using a much smaller budget than AF-MCTS.

Generating alternatives without trained rewards or historical memory. PARM and SemOPT guide search with learned rewards, while DCM-Agent uses memories of historical solutions. Their effectiveness depends partly on the quality and coverage of those examples. DiverseForm explores alternative interpretations without additional training or memory of historical solutions. These candidates could provide valuable positive examples for reward-model training or enrich modeling memories.

Q2. What counts as correct under underspecification?

A2. DiverseForm is intended as a formulation assistant that presents a small set of semantically valid formulations for human inspection. Our Pass@5 metric measures a narrower property: whether at least one candidate produces an interpretation that matches the intended problem.

However, we agree that reliable candidate selection remains important beyond the numerical agreement. As a supplementary analysis, we add the LLM defensibility evaluation using GPT-6-Astra. The result shows that an LLM judge applies conservative screening to flag defensibility, preserving 87.9% of reference-matching candidates for human review. Conversely, 4.2% of candidates judged defensible did not match the reference, potentially reflecting valid alternative interpretations or evaluator errors. We therefore explore this assessment as a potential aid to human inspection, with independent expert validation left to future work.

Q3. Why exploration gains vary across benchmarks and models

A3. We thank the reviewer for highlighting this variation. Here are some insights:

First, models may differ both in how strongly independent samples concentrate on one interpretation and in how successfully they follow the request to produce a defensible alternative. A model that already covers several interpretations independently has less to gain; a model that repeatedly chooses one reading but can generate alternatives when prompted may benefit more.

Second, benchmark size affects the apparent magnitude of changes. On ComplexOR, with 17 retained problems, one additional success changes a single run’s accuracy by approximately 5.9 percentage points. Large percentage changes on this benchmark should therefore be interpreted alongside the underlying counts and run-to-run variability.

Dear Reviewer FkmE,

Thank you again for your constructive feedback and for taking the time to review our paper. As the discussion period is nearing its end, we wanted to follow up to see whether our responses, new experimental results, and paper revisions have addressed your main concerns. Please let us know if any points remain unclear or if further clarification would be helpful. We would be very glad to discuss further.

These are the new/revised experimental results:

  • MCTS-based search method as a baseline
  • LLM evaluation of candidates over diverse prompting
  • New problem set for ambiguity-injection experiment with non-trivial ambiguity cases

Best regards,

The Authors

I thank the authors for the great effort in providing additional explanations and results.

Regarding the variation of exploration gains across benchmarks and models, I think the authors' explanation makes sense. The results across models are consistent: Fable 5 and GPT-6-Astra are better models and therefore have smaller gains. There is also randomness due to the use of LLMs. So I would buy the explanations.

However, my concern remains how is the proposed framework different from existing search-based methods. The authors cited several explanations:

Direct declarative generation without intermediate modeling stages. To me, this seems to be the result of better capabilities of new models (Sonnet, etc.). I doubt if open-source, low-capability models (e.g., GPT-OSS) can come up with complete models in one shot. History-conditioned exploration of alternative interpretations. Again, MCTS can be considered as history-conditioned exploration, because each node expansion depends on the path from the root to the parent node (i.e., history). Generating alternatives without trained rewards or historical memory. This is a little contradictory to the previous point...

In summary, I would like to maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.