- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DiverseForm, a training‑free formulation assistant that translates natural‑language optimization problems into mathematical formulations under underspecification. DiverseForm employs history‑conditioned diverse sampling: each subsequent query incorporates earlier encodings along with their solver statuses, prompting the model to propose a different defensible interpretation of the problem description. A declarative intermediate representation (IR) is adopted to explicitly capture variables, constraints, and objectives, thereby surfacing differences among candidate formulations. Evaluated on four optimization‑modeling benchmarks (NL4Opt, MAMO ComplexLP, IndustryOR, and ComplexOR) using Sonnet 4.6 and Gurobi, DiverseForm achieves 97.4% overall Pass@5, outperforming all prompting‑based and agentic baselines. In the open‑model study, Qwen3‑14B with thinking enabled surpasses the task‑specifically fine‑tuned LLMOPT in overall accuracy without additional training. Further analysis covers ambiguity‑injection recovery, lexical versus numerical diversity, and case studies of interpretation categories.
Critical strengths
This work identifies a practically important and under-explored failure mode of LLM-based auto-formulation: executable but wrong formulations caused by underspecified natural language, which escape execution-based debugging. The proposed history-conditioned exploration yields substantial coverage gains (from 89.2% to 97.4% overall Pass@5 with Sonnet 4.6), outperforming the strongest baseline by 5.1 percentage points with fewer than half as many model calls (5.0 versus 11.5). The accompanying diagnostic studies (interpretation categories, ambiguity injection, lexical-versus-numerical diversity) provide useful insights into where problem descriptions require clarification, which may inform future research on clarification-oriented formulation assistants.
Critical weaknesses
-
The paper exhibits limited conceptual novelty and largely resembles a solid prompt-engineering study tailored for the auto-formulation setting. The core mechanism, conditioning each new sample on previous generations and instructing the model to take a different defensible interpretation, is a direct application of well-known diversity-promotion ideas from recent literature (repeated sampling in self-consistency, conditioning on previous responses in integrative decoding, and guided diversity in G2). The exploration is also heavily scaffolded: the sampling prompt in Appendix B.2 itself enumerates six kinds of interpretation decisions for the model to take differently, so candidates largely vary along pre-specified dimensions rather than discovered ones. The declarative IR, the second pillar, is closely related to the meaning representations of NL4Opt and the Canonical IR, and its own ablation shows a modest 0.9-point overall gain for the frontier model, with executable code even surpassing the IR on IndustryOR (95.7 versus 93.8); the benefit is concentrated in the smaller model (+4.4 points for Qwen3-8B). Notably, one-shot enumeration with an interpretation guide matches DiverseForm’s overall coverage (97.4%) in a single call (Table 4). It remains hard to disentangle how much of the improvement comes from the proposed iterative exploration paradigm versus simply drawing more samples with an explicit diversity instruction.
-
The experimental evaluation has several notable drawbacks. (1) The headline metric is oracle Pass@5 under a 5% numerical tolerance. Numerical agreement does not establish formulation correctness (as the paper itself acknowledges), and no candidate-selection mechanism is proposed, so the reported coverage overstates what a deployable formulation assistant could deliver. The metric also conflates resolving genuine underspecification with guessing the annotator’s reading: the paper’s own case studies show that references impose a stated preference in one problem but omit it in another (NL4Opt 143 versus 202). In addition, all headline numbers are computed on an adjudicated 495-instance subset after excluding 79 instances, whose documentation is itself inconsistent: Table 6, captioned as complete, lists only 71 exclusion identifiers (e.g., 16 of the 22 NL4Opt exclusions, and no IndustryOR entry despite one exclusion in Table 5). (2) The headline comparison runs each baseline at its own call budget (1.7 to 31.4 calls per problem), so it conflates the exploration strategy with workflow design. The paper’s budget-controlled comparisons tell a weaker story than the headline 5.1-point margin: under the same five-proposal budget, independent sampling trails history-conditioned sampling by only 2.8-4.9 points (Figure 4), and guided one-shot enumeration matches the overall coverage in a single call, though that call may emit an uncapped number of candidates evaluated under Pass@any, and its interpretation guide was derived from the recovered cases, i.e., evaluated in-sample (Appendix B.4), with no held-out validation reported. (3) The ambiguity-injection study is small and constructed: 59 variants are derived from 56 problems of the elementary NL4Opt benchmark, the construction only retains variants whose alternative reading changes the optimum (a selection bias toward numerically observable ambiguity), only one model is tested, and only recovered cases are analyzed, with no inspection of the problems exploration still misses. The reported numbers are also internally inconsistent: Section 4.5 reports 59 variants with 16 independent-sampling misses and 15 recoveries, whereas Appendix E.1 describes 26 edited descriptions with 8 misses, all recovered, and the appendix further admits that raw traces were unavailable in its snapshot, leaving the recovery counts not trace-verifiable. (4) No variance or statistical significance is reported for any comparison despite sampling at temperature 0.7. The margins are sometimes razor-thin: on ComplexOR, with only 17 retained instances, the reported advantage over Standard and OptiTree (88.2% versus 82.4%) corresponds to exactly one problem, so it is unclear whether the claimed per-benchmark wins are stable across runs.
Questions for authors
-
Figure 4 compares independent sampling with history-conditioned diverse sampling, but the two conditions differ in both the displayed history and the explicit diversity instruction. Can you isolate the contribution of history conditioning itself (e.g., history without the diversity instruction, or the instruction without history)? Relatedly, since the interpretation guide was derived from the recovered cases (in-sample, per Appendix B.4), how does guided one-shot enumeration perform on held-out problems or unseen interpretation categories?
-
DiverseForm leaves candidate selection to human review, yet all headline results are reported as oracle Pass@5. How should the final formulation be selected in practice, and what is the end-to-end accuracy under a realistic non-oracle selection mechanism? Relatedly, could you reconcile the discrepancy between Section 4.5 (59 variants, 16 independent-sampling misses, 15 recoveries) and Appendix E.1 (26 edited descriptions, 8 misses, all recovered)?
2 comments
Dear Reviewer s8FR,
Thank you again for your constructive feedback and for taking the time to review our paper. As the discussion period nears its end, we wanted to follow up to see whether our responses, new experimental results, and paper revisions have addressed your main concerns. Please let us know if any points remain unclear or if further clarification would be helpful. We would be very glad to discuss further.
These are the new/revised experimental results:
- Main result of Diverseform without any instruction guidance
- MCTS-based search method as a baseline
- LLM evaluation of candidates over diverse prompting
- New problem set for ambiguity-injection experiment with non-trivial ambiguity cases
Best regards,
The Author
Q1. Novelty beyond existing diversity prompting and intermediate representations.
A1. We agree that neither structured representations nor history-conditioned generation is new on its own. Our contribution lies in how we combine these components to generate and explore different mathematical interpretations of the same natural-language(NL) problem. Together, these findings show how declarative generation and diverse sampling can improve accuracy while making unresolved assumptions visible for human review.
Building on the structured representation established by NL4Opt and Canonical IR, we contribute a concrete generation procedure using declarative syntax in the prompt, enabling the LLM to encode mathematical formulations directly from the NL description. Our syntax-based prompting significantly reduced failures due to execution-based errors on both large and small models compared to the other baselines.
Sonnet 4.6 (Figure 2)
Qwen3-8B with thinking disabled (Table 8)
Combined with the declarative representation, our method promotes exploration in the structured semantic space, making resulting modeling choices explicit and comparable, unlike G2, which promotes diversity through token-level decoding interventions informed by prior outputs. The code-versus-IR ablation indicates that the representation complements this exploration.
Q2. Limits of oracle Pass@5 and candidate selection without reference answers.
A2. DiverseForm is intended as a formulation assistant that presents a small set of semantically valid formulations for human inspection. Our Pass@5 metric measures a narrower property: whether at least one candidate produces an interpretation that matches the intended problem.
However, we agree that reliable candidate selection remains important beyond the numerical agreement. As a supplementary analysis, we add the LLM defensibility evaluation. The result shows that an LLM applies conservative screening to flag defensibility, preserving 87.9% of reference-matching candidates for human review. Conversely, 4.2% of candidates judged defensible did not match the reference, potentially reflecting valid alternative interpretations or evaluator errors. We therefore explore this assessment as a potential aid to human inspection, with independent expert validation left to future work.
Q3. Separating history conditioning from explicit diversity instructions and predefined categories.
A3. We thank the reviewer for identifying the interpretation scaffolding in the original prompt. We removed the six predefined interpretation categories from Appendix B.2 and reran the main evaluation using the revised prompt. DiverseForm now achieves 98.5% overall reference-answer coverage with Sonnet 4.6, compared with the previously reported 97.4%.
For Figure 4, we rerun three frontier models with the revised prompt to evaluate the contribution of history conditioning compared with independent sampling. It improves coverage over independent sampling for all three frontier models. These results show that the gains do not depend on supplying the six predefined interpretation categories.
Q4. Guided enumeration as a competing baseline: candidate budgets and held-out generalization.
A4. This experiment shows why we chose successive generation of new candidates through iterative calling, adding prior outputs to the prompt. The revised results in Table 4 show that DiverseForm achieves 98.5%, compared with 95.2% for unguided enumeration and 97.4% for guided enumeration. Enumeration uses one model call to enumerate at most five encodings. The result also shows that history conditioning is better at finding possible interpretations, even without interpretation guidance, than case-derived interpretation guidance during enumeration.
Q5. Fair call-budget comparisons and statistical reliability of reported gains.
A5. We have clarified that the “Calls per candidate” column in Table 1 reports the number of LLM invocations required to produce one complete candidate. To assess sensitivity to stochastic generation, we repeat each sampling condition in Figure 4 three times for each model and report the mean Pass@5.
Q6. Scope, selection bias, and inconsistent reporting in the ambiguity-injection study.
A6. We acknowledge the study’s limited scope. We formed new ambiguity-injection cases across multiple benchmarks, verified to preserve the original reference optimum while introducing an alternative interpretation with a different optimum. Diverse sampling recovered all six cases that independent sampling missed, supporting its ability to explore interpretations overlooked by independent sampling. We reported these six cases in Appendix F and updated the results in Section 4.5 and Figure 5.