- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies pre-formulation clarification in LLM-based optimization modeling. It introduces OR-Clarify, a benchmark containing 100 incomplete business specifications with hidden requirements, and InterOPT, a two-stage framework that tracks formulation gaps and guides clarification questions and stopping decisions. Experiments show substantial improvements in choice-based requirement recovery, although the method is less competitive in open-ended interactions.
Critical strengths
S1. A meaningful and practically relevant problem.
Recognizing missing requirements before constructing an optimization model is an important but underexplored capability. The paper usefully distinguishes formulation-critical ambiguities from ordinary missing details.
S2. Structured evaluation and informative diagnostics.
OR-Clarify provides hidden-slot annotations, controlled user simulation, and separate Open and Choice protocols. The Choice-setting improvements are substantial, and the stopping analyses, question-budget curves, and supplementary implementation provide useful evidence for evaluating clarification behavior.
Critical weaknesses
W1. The benchmark does not fully validate the central claim of formulation readiness. [Q1]
Section 3 defines completeness in terms of whether plausible business completions induce the same mathematical formulation. However, the evaluation primarily measures whether agents explicitly ask about predefined hidden slots, without verifying that the resulting specification supports a correct formulation.
Furthermore, all benchmark cases contain missing requirements, so the ability to recognize an already-complete specification and stop appropriately remains untested. This creates a gap between the proposed completeness criterion and the actual evaluation.
W2. The gap-tracking and stopping mechanisms are not sufficiently justified. [Q2]
The Choice implementation marks gaps as ASKED immediately after question selection, without explicitly tracking whether the user's response resolves them. The supplementary materials also reveal differences between the Open and Choice implementations that are not adequately explained in the unified method description.
These concerns are reflected in the Open results: InterOPT underperforms ORPilot in Core Exact (0.538 vs. 0.583), while producing more silent assumptions (0.560 vs. 0.268 per run). Removing Stage 2 further reduces silent assumptions to 0.254. The evidence therefore provides limited support for the reliability of the proposed stopping mechanism.
W3. The stage ablations do not adequately isolate the proposed contributions. [Q3]
The supplementary results reveal substantial differences in question budgets. In Choice, full InterOPT asks 10.04 atomic questions on average, compared with 3.46 and 4.86 for the two stage ablations.
Although the paper reports question-budget curves for the main baselines, equivalent controls are missing for these ablations. Thus, it remains unclear how much of the observed improvement comes from the proposed mechanisms rather than additional interaction.
The absence of complete benchmark cases and case-level evaluation artifacts also limits independent verification of the reported results.
Questions for authors
[Q1] Formulation completeness and benchmark validity.
Could the authors include already-complete specifications to evaluate unnecessary questioning and premature stopping? More importantly, could they verify whether recovered requirements lead to structurally correct mathematical formulations, rather than only successful hidden-slot matching?
[Q2] Gap resolution and stopping reliability.
Why does the explicit ledger track ASKED rather than answer-conditioned resolution? How are ambiguous or incomplete answers handled? Could the authors also clarify the differences between the Open and Choice implementations and explain why removing Stage 2 substantially reduces silent assumptions in Open?
[Q3] Fair isolation of the proposed stages.
Could the authors provide question-budget-matched comparisons between full InterOPT and its stage ablations? A corresponding LLM-call or token-budget comparison would also help establish whether the gains arise from the proposed mechanisms rather than additional deliberation.
6 comments
Response to Reviewer TM6R - Part2
W2.2 & Q2.1 & Q2.2: Answer-Conditioned Resolution & Ambiguous User Answers
First, we thank you for this insightful suggestion, which motivates us to introduce an answer-conditioned variant (InterOPT-Resolution) that explicitly checks whether a gap is resolved and follows up on the same gap when the user's response is insufficient. Specifically, at the next planning step, InterOPT-Resolution classifies each queried gap as open, partially resolved, or resolved based on the user's response. More details are provided in our revised paper.
Second, your suggestion also motivates us to introduce an additional evaluation protocol, where the simulated user provides an incomplete or underspecified answer to a clarification question with a predefined probability. This protocol better captures a practical challenge in real-world interactions: users, particularly non-experts, may not fully understand what information a question seeks and may therefore provide only partial answers. More details are provided in our revised paper.
We evaluate this protocol under both Open and Choice settings and report the results below:
| Setting | Method | All-Slot Exact ↑ | Core Exact ↑ | Silent/run ↓ |
|---|---|---|---|---|
| Open | ORPilot (Best Baseline) | 0.500 | 0.536 | 0.467 |
| Open | InterOPT | 0.533 | 0.607 | 0.167 |
| Open | InterOPT-Resolution | 0.567 | 0.607 | 0.133 |
| Choice | ReadyGate (Best Baseline) | 0.467 | 0.500 | 0.533 |
| Choice | InterOPT | 0.633 | 0.643 | 0.167 |
| Choice | InterOPT-Resolution | 0.633 | 0.679 | 0.333 |
From the results, we draw two observations:
- Answer-conditioned resolution improves requirement recovery under incomplete user answers. It improves All-Slot Exact in Open and Core Exact in Choice, suggesting that explicitly tracking gap resolution and following up on insufficient answers helps recover missing requirements, although its effect on silent assumptions varies across protocols.
- ORPilot is less robust to incomplete user answers. Despite its strong performance under Open settings with complete user answer in the original paper, its recovery performance drops and silent assumptions increase under incomplete answers. We attribute this partly to its tendency to ask broad invitation questions (e.g., "Is there anything else we should know?"), which may elicit uncertain or incorrect information in this setting. Further analysis is provided in our response to Reviewer 7gVL (Point 2).
W2.3 & Q2.4: Stage 2 ablation under Open settings w.r.t. Silent Assumptions
Your observation highlights an important trade-off in Stage 2. Stage 2 uses gap information from Stage 1 and candidate questions to select the most informative question at each turn, rather than asking about every possible detail. As a result, some lower-priority details with natural default values may remain unconfirmed. If the agent later treats these defaults as confirmed in its final summary, they count as silent assumptions.
We illustrate this with Case 59 from InterOPT-Open, which involves monthly production planning for three candy brands using three raw materials:
- Turn 2: After the user confirms profit maximization in Turn 1, the agent considers asking whether profit per kg is calculated as "selling price minus raw material cost minus processing fee."
- Question prioritization: Instead, it asks about the missing composition constraints for brand B, reasoning that "The profit calculation detail is important but has a natural default interpretation." It then asks about production bounds in Turn 3 and stops.
- Turn 4: In its
READY TO MODELresponse, the agent applies the profit formula without asking the user to confirm it. The judge therefore records one silent assumption (H1), since the agent treats the formula as confirmed without checking whether both raw material costs and processing fees are subtracted.
Based on this case and the ablation experiment, we can see that Stage 2 enables fewer, more targeted questions, although some assumptions with natural default interpretations may remain unconfirmed. By prioritizing the most informative questions, Stage 2 reduces atomic questions (4.82 vs. 9.50) and improves Core Exact (53.8% vs. 50.6%). However, some details, such as the profit calculation formula in Case 59, may remain unconfirmed and become silent assumptions in the final summary, leading to a higher Silent/run (0.560 vs. 0.254). Nevertheless, this does not come at the cost of overall requirement recovery and downstream modeling accuracy, as InterOPT still achieves higher All-Slot Exact and Core Exact.
Response to Reviewer TM6R - Part3
W3.1: Fine-grained ablation w.r.t. turn budgets
To address your concerns about the ablation under the Choice setting, we conduct a fine-grained evaluation of full InterOPT, its two ablation variants, and the strongest baselines under controlled interaction budgets of up to 3, 4, and 5 turns per OR-Clarify case. The results are reported below:
| Interaction Budget | Metric | Full InterOPT (Choice) | w/o Stage 1 (Choice) | w/o Stage 2 (Choice) | ReadyGate (Choice) | ORPilot (Open) |
|---|---|---|---|---|---|---|
| Up to 3 turns | Core Exact ↑ | 54.47% | 53.19% | 49.36% | 47.45% | 49.79% |
| All-Slot Exact ↑ | 51.00% | 48.80% | 46.20% | 44.00% | 46.80% | |
| Up to 4 turns | Core Exact ↑ | 58.09% | 55.11% | 53.83% | 49.57% | 52.77% |
| All-Slot Exact ↑ | 54.00% | 50.80% | 49.80% | 46.00% | 49.60% | |
| Up to 5 turns | Core Exact ↑ | 60.64% | 55.96% | 56.60% | 50.85% | 55.96% |
| All-Slot Exact ↑ | 56.00% | 51.60% | 52.40% | 47.00% | 52.20% | |
| Full trajectory (20-turn limit) | Core Exact ↑ | 67.45% | 56.38% | 61.49% | 51.70% | 58.30% |
| All-Slot Exact ↑ | 63.80% | 52.00% | 57.20% | 47.60% | 54.60% |
From the table above, we draw three observations:
-
Larger interaction budgets benefit all methods. Both All-Slot Exact and Core Exact improve as the interaction budget increases, as additional turns allow agents to recover more hidden requirements from user responses.
-
InterOPT consistently outperforms the ablations and baselines under the same budget. This suggests that its improvements do not simply come from having more interaction turns, but from our core clarification strategy.
-
InterOPT makes more effective use of additional interaction turns. Its performance advantage becomes more pronounced as the budget increases. This suggests that InterOPT better exploits additional interaction opportunities, while other methods would experience diminishing returns in later turns due to less targeted or repetitive questions.
W3.2: LLM API Calls & Token Costs
Following your suggestion regarding the computational cost of InterOPT and its variants, we further report the average number of LLM API calls and input/output tokens per question turn, together with the corresponding requirement-recovery performance:
| Setting | Method | All-Slot Exact ↑ | Core Exact ↑ | API Calls per Question Turn ↓ | Input Tokens per Question Turn ↓ | Output Tokens per Question Turn ↓ |
|---|---|---|---|---|---|---|
| Choice | w/o Stage 1 | 52.0% | 56.4% | 4.98 | 11,702 | 8,921 |
| Choice | w/o Stage 2 | 57.2% | 61.5% | 3.66 | 7,824 | 5,527 |
| Choice | InterOPT (ours) | 63.8% | 67.5% | 5.38 | 18,819 | 10,169 |
As expected, using both Stage 1 and Stage 2 incurs more LLM calls and token usage than either ablation variant. However, this additional computational cost comes with substantially better requirement recovery: InterOPT achieves the highest All-Slot Exact and Core Exact. Together with our downstream modeling evaluation, these results suggest that the additional computation is worthwhile, as it helps recover more formulation-critical requirements and supports more accurate downstream modeling.
References
[1] OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models.
Response to Reviewer TM6R - Part1
Dear Reviewer TM6R,
Thank you for recognizing the practical relevance of our problem and the value of our structured evaluation, particularly the improvements under Choice. Below, we provide additional explanations and experimental evidence addressing your concerns regarding formulation readiness, stopping decisions, and stage ablations.
W1.1 & Q1.2: Downstream Mathematical Modeling Accuracy
Following your insightful suggestions, we compare four clarification methods end-to-end on our proposed OR-Clarify, using OptiMUS [1] as the downstream automated modeling framework. The public problem statements together with the information elicited during the clarification dialogue are combined as inputs for OptiMUS. We report both the clarification metrics (i.e., All-Slot Exact and Core Exact) and the final modeling accuracy as follows:
We can observe that our InterOPT-Choice achieves the highest final modeling accuracy. This demonstrates that a better elicitation strategy can benefit the downstream optimization modeling performance.
W1.2 & Q1.1: Stopping on Initially Complete Specifications
Thank you for your constructive suggestions! We further evaluate InterOPT and the strongest baseline under both Open & Choice settings, using the complete specifications corresponding to the 100 OR-Clarify cases, without informing the methods that the specifications are already complete. We impose a 20-turn interaction limit per problem and report the following three metrics:
READY TO MODELwithout asking any clarification questions.READY TO MODELor reaches the turn limit. Immediate stops count as zero.READY TO MODEL.Overall, InterOPT requires fewer clarification turns and reaches the turn limit less frequently than the strongest baseline under both settings. This suggests its effectiveness in avoiding unnecessary questions and recognizing when to stop.
Your suggestion highlights another important and underexplored challenge in pre-formulation clarification for OR: determining when to stop asking questions and proceed to optimization modeling, particularly when the initial specification is already complete. We appreciate this insight and include this additional evaluation to examine whether clarification agents can recognize complete specifications and avoid unnecessary questioning.
W2.1 & Q2.3: Open/Choice Implementation
In the Choice setting, the agent presents structured alternatives, and the user selects among predefined options or selects Choice D instead to provide supplementary information. In the Open setting, the agent asks free-form questions, and the user responds in natural language. Both protocols share the same two-stage clarification logic, using formulation gaps to guide question selection. We clarify these differences in Section 5 of the revised manuscript.