acceptodds
Withdrawn from ICLR 2027

Evaluating LLM-Driven Interactive Optimization Modeling: A Knowledge-Grounded Benchmark for Conversational Specification Recovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Recent studies have explored the capability of Large Language Models (LLMs) for formulating and solving optimization problems. Real-world optimization modeling, however, often starts from vague and incomplete operational descriptions that must be clarified through interaction. In this work, we study whether LLMs can act as modeling experts and recover complete optimization specifications through multi-turn dialogue. We formulate this process as conversational specification recovery, where an LLM must identify missing requirements, ask questions, and reconstruct a complete problem under a fixed dialogue budget. To evaluate this capability, we introduce OptRecover, a knowledge-grounded benchmark with an integrated evaluation harness that combines reference-grounded answering from the hidden complete specification, semantic parsing of natural-language descriptions, and deterministic scoring of the resulting semantic gaps. A shared structured representation of optimization knowledge connects these components and enables controlled information removal and fine-grained evaluation. We instantiate the benchmark on matching and assignment problems, comprising 48 complete problems, 288 systematically constructed incomplete variants, and 336 structured ground-truth specifications. Our evaluation reveals substantial differences in specification recovery across LLMs and examines how performance varies across semantic dimensions, problem complexity, and missingness levels.

Primary area:
datasets and benchmarks
Keywords:
Combinatorial Optimization, Model Formulation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_jFKtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies if a model can recover all the information to get a complete spec using conversations. They introduce a benchmark, OptRecover, to perform this task. They evaluate on latest models and introduce Knowledge-grounded representation as well as a harness for performing the task. Findings include that current models cannot do this task reliably.

Critical strengths

I really like the idea of the work. Humans are usually vague in their specifications. Most of the progress is made by LLMs on math tasks because they are well specified. Hence this task of seeing if the LLMs can recover the correct specification is super interesting and useful.

The work evaluates on several new methods and tries to evaluate different stages using a fixed representation (although this could be a drawback as well)

Critical weaknesses

There are missing details in the draft on how the benchmark was constructed. such as: who are the experts who designed the optimization questions?

The gold answers were bootstrapped from the same parser that later scores model outputs.

There is no information on inter-annotator agreement and no count of how much experts changed the parser drafts.

The paper simulates users using LLMs, it would be better to have some human evaluation.

I strongly believe that the paper is very narrow although it had a potential to evaluate something better.

The measurement itself isn’t validated well enough to trust the rankings of the models. there are no statistics. Since the users are simulated, do we know how well are do they answer? If they even answer correctly?

There is no evaluation on how the scores change with different budget. Since the paper claims that they can do this at a budgeted setting, knowing this is super important.

The paper is also very hard to read with a lot of things blocks etc.

Questions for authors

Why is the natural language quesiton only based on x0x_0 shouldn't it be based on all the collected information? i.e. all question and answer pairs? I would imagine that the answer to previous question would change what i need to know next?

Why do we have business experts? you didn't evaluate with them either.

You say consolidated specification is reviewed by AI reviewer but how does the AI reviewer know if this is complete or not?

Reviewer_mS8vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OptRecover, a knowledge-grounded benchmark for evaluating whether LLMs can act as modeling experts and recover complete optimization specifications through multi-turn dialogue. The benchmark is built on matching and assignment problems and comprises 48 complete problems, 288 systematically constructed incomplete variants, and 336 structured ground-truth specifications, organized into four difficulty levels (L1–L4). An integrated evaluation harness combines reference-grounded answering, semantic parsing of natural-language descriptions, and deterministic scoring of the resulting semantic gaps, all connected by a shared structured representation of optimization knowledge.

Experiments on Claude, GPT, and DeepSeek reveal substantial differences in specification-recovery performance across models, semantic dimensions, problem complexity, and missingness levels.

Critical strengths

The problem studied, conversational specification recovery under a fixed dialogue budget, is both interesting and practically relevant to real-world optimization modeling, where descriptions are often vague and incomplete.

The structured annotations and the controlled information-removal design provide a valuable reference for future work in this direction.

Critical weaknesses

The scoring framework, which uses C and M to denote the weights of expressed and missing/incorrect reference components respectively, is not sufficiently justified. Assigning weights to components of a structured optimization specification is inherently subjective: different parts (e.g., the objective function, individual constraints, decision variables) may carry very different importance, yet the paper does not explain how these weights are determined or validated. A sensitivity analysis or a discussion of alternative weighting schemes would strengthen this aspect.

The definition of C and M should be made more explicit. In particular, how is a component deemed "expressed" versus "missing or incorrect" after semantic parsing, and what happens when a component is only partially correct?

The evaluation procedure may be sensitive to surface-level phrasing differences. LLMs of different families often produce semantically equivalent answers in different wording, so a matching-based scoring step risks under-scoring correct recoveries or over-scoring partially wrong ones. The authors should include concrete examples that illustrate such mismatches, report the resulting error rate, and discuss how much the reported rankings are affected.

Questions for authors

I would like to see examples of the real dialog and how they are evaluated as demonstration and analysis for the evaluation procedure.

I do think the effectiveness of evaluation is the key to the success of this benchmark. It would be better to provide more evidence in this direction.

Reviewer_YTS3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OptRecover, a benchmark for testing whether LLMs can recover missing requirements in optimization problems through conversation. Starting from an incomplete description, a model asks up to 15 clarification questions and then writes a complete specification. Answers come from a simulated user grounded in the hidden reference problem. The benchmark contains 48 complete matching and assignment problems across 12 scenarios, with 288 incomplete variants covering different levels of complexity and missing information. Its main contributions are the expert-reviewed dataset, a structured representation of optimization requirements, and an evaluation system that combines reference-grounded answering, LLM-based parsing, and deterministic scoring.

Critical strengths
  • Missing requirements such as whether a fee applies per job or per worker, can change the intended optimization problem. Evaluating whether models ask the right clarification questions and preserve the answers captures an important requirement for useful modeling assistants. This gives the benchmark clear practical value.
  • Expert-reviewed complete problems are paired with deliberately incomplete variants across several complexity and missingness levels. The shared representation allows the scorer to identify missing requirements, altered meanings, and unsupported additions. This provides a systematic way to evaluate whether a conversation improves the specification while preserving information already given.
  • The answerer retrieves relevant facts from the hidden specification before expressing them in natural language, and reports unknown details when the reference does not resolve them. This is an important design choice for an interactive benchmark: controlling what is revealed helps ensure that recovery reflects the model’s questioning strategy. It also makes the information supplied during each dialogue easier to audit.
Critical weaknesses
  • Parser validation covers complete descriptions, not imperfect recovered outputs. Selecting the best of three parses against the reference may inflate scores. Human checks and single-parse results are needed to establish reliability. -There is no comparison with reconstruction without dialogue, a fixed checklist of questions, or different question budgets. These controls would help establish whether recovery depends on effective questioning and how much interaction is needed. The current results mainly rank models by their final outputs, leaving the paper’s central conversational capability insufficiently examined.
  • The appendix states that outputs that cannot be scored are excluded, but the results do not report these counts for each model. If exclusion rates differ, models could be compared on different subsets, with some failures left out. This matters because the reported scores should be accompanied by evidence that the evaluation covers comparable instances.
  • SRR measures improvement relative to the initial gap, so a small remaining error can produce a low recovery rate when little information was initially missing. The authors acknowledge this effect, but still conclude that mild omissions are harder to identify. Absolute initial and final scores, together with an analysis of which requirements were recovered, are needed to support that conclusion.
Questions for authors
  • Were actual dialogues and recovered descriptions checked by humans? How do scores and rankings change when using a single parse instead of the best of three?
  • How many instances were excluded for each model, and were models compared on the same set of instances?
  • Do you have results for reconstruction without dialogue or a fixed question checklist that help establish the benefit of adaptive questioning?
  • What evidence from absolute scores or individual recovered requirements supports the claim that mild omissions are harder to identify, beyond the effect of SRR normalization?
  • How often did the simulated user reveal requirements that were not requested, omit relevant details, or give unsupported answers? Were these behaviors independently checked?
  • Can the existing dialogues distinguish requirements that were never elicited from requirements that were correctly disclosed but lost or changed in the final specification? This would help separate questioning failures from consolidation failures.
Reviewer_VZyYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether an LLM can act as a modeling expert and recover a missing optimization specification through conversation. The model starts from an incomplete description, asks clarifying questions to a grounded answerer under a fixed budget, and then merges the original text with the dialogue into one complete specification. The authors contribute (i) OptRecover: 48 expert-built matching/assignment problems in four complexity levels, expanded into 288 incomplete variants at three missingness levels, plus 336 structured goldens and a shared representation (problem structure, 14 mechanism Blocks, indexed Families, grounded parameters); (ii) a harness that combines reference-grounded answering, an LLM parser with voting consensus, and a deterministic slot-level scorer reporting SRR and five diagnostics; (iii) experiments with six LLMs (one dialogue per instance, up to 15 questions) and a parser-backbone study on the 48 complete problems.

Critical strengths
  • The task is genuinely new and well motivated. Prior NL-to-formulation work assumes the description is already complete and only tests formalization. Here the missing content — charging basis, activation, scope — is itself the object of study, and the paper argues convincingly that it cannot be read off the numbers. The framing is positioned well against both formalization and clarification benchmarks.
  • The construction and scoring are unusually auditable. Requirements are removed at the level of semantic facts, with surrounding text rewritten so the omission does not leak, and variants are cross-checked by several OR experts against both a complete and an initial golden. Scoring is a deterministic per-slot ledger (C/M/E) over a representation that keeps scope, activation vs. applicability, event frequency, units and partial sets apart. This goes well beyond LLM-judge scoring and allows fine-grained error analysis.
Critical weaknesses
  1. No baseline isolates the value of interaction, and there is no human reference point, so the headline number cannot be interpreted. The obvious controls are missing: the same model consolidating the initial text with no dialogue, an oracle setting where the removed facts are handed over directly, and a human OR expert on a small sample. Without the first, I cannot tell how much of the 66.14% comes from the answers and how much from the consolidation prompt's own priors about what an assignment problem usually looks like. Without the human number, there is no scale for whether 66% is good or poor. Since dialogue-driven recovery is the paper's whole claim, this is the decisive gap.
  2. Parse selection uses the hidden reference and everything is single-sample, so the comparative claims are not established. Choosing the best of three parses by agreement with the complete golden makes every SRR a best-case upper bound, and it quietly favours models whose outputs vary more, since they get more draws. With one dialogue per instance, no confidence intervals, and only 48 underlying problems behind 288 non-independent variants, differences such as 45.05 vs. 47.77 vs. 52.05, and the trends in Tables 3–4, cannot be separated from noise. The authors are open about the best-of-three choice in Appendix C.1, but they do not address the consequence.
Questions for authors
  1. What SRR do the six models get with no dialogue at all, and with an oracle that supplies the removed facts verbatim? Reporting the interaction-only gain would make the central claim testable — without it, how should a reader attribute Table 2?
  2. How does the ranking change if you report the mean of the three parses, or a single pre-registered parse, instead of the reference-selected best-of-three? Does the ~66% hold, and does the model order in Tables 2–4 survive?
  3. How was answerer correctness validated, and what share of answers are wrong, incomplete, or wrongly reported as unknown? Can you split lost SRR into questioner error versus answerer/routing error?
  4. What SRR does a human OR expert achieve on a sample of the 288 variants, and what is inter-annotator agreement on the goldens? Relatedly, what ceiling should we assume, given that the parser scores recovered text rather than the formulation itself?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.