Evaluating LLM-Driven Interactive Optimization Modeling: A Knowledge-Grounded Benchmark for Conversational Specification Recovery

Recent studies have explored the capability of Large Language Models (LLMs) for formulating and solving optimization problems. Real-world optimization modeling, however, often starts from vague and incomplete operational descriptions that must be clarified through interaction. In this work, we study whether LLMs can act as modeling experts and recover complete optimization specifications through multi-turn dialogue. We formulate this process as conversational specification recovery, where an LLM must identify missing requirements, ask questions, and reconstruct a complete problem under a fixed dialogue budget. To evaluate this capability, we introduce OptRecover, a knowledge-grounded benchmark with an integrated evaluation harness that combines reference-grounded answering from the hidden complete specification, semantic parsing of natural-language descriptions, and deterministic scoring of the resulting semantic gaps. A shared structured representation of optimization knowledge connects these components and enables controlled information removal and fine-grained evaluation. We instantiate the benchmark on matching and assignment problems, comprising 48 complete problems, 288 systematically constructed incomplete variants, and 336 structured ground-truth specifications. Our evaluation reveals substantial differences in specification recovery across LLMs and examines how performance varies across semantic dimensions, problem complexity, and missingness levels.