- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to use decisions from an existing optimization model and user-provided target decisions to correct missing or misspecified constraints, objectives, and variables. TACIT combines two approaches: a bottom-up approach that generates cuts through optimization and uses an LLM to contextualize and generalize them into meaningful constraints, and a top-down approach in which an LLM proposes structural modification templates whose parameters are determined through optimization. Candidate transformations are evaluated using objective calibration and separation loss, and the incumbent formulation is updated iteratively. The authors evaluate TACIT on 38 scenarios spanning nine classes of optimization problems, with target decisions generated by a predefined target model for each scenario. With GPT-5.6-sol, TACIT achieves an average recovery rate of 64.2% across five runs, compared with 46.8% for the LLM-only baseline. The proportion of scenarios recovered in at least one of the five runs is 78.9%, compared with 60.5% for the baseline.
Critical strengths
- The problem has practical value. Deployed optimization models often require manual corrections, but these corrections usually appear as changes to decisions rather than explicit explanations of what is wrong with the formulation. The paper uses this feedback to repair the model, including cases that require new auxiliary variables, constraints, and objective terms. This allows it to address structural misspecifications that cannot be resolved by adjusting existing parameters alone.
- The recovery criterion is demanding. The repaired model must produce solutions that are feasible under the target model and within 1% of its optimal objective value on all 256 unseen test instances, rather than merely fit the observed corrections. The experiments cover different types of formulation errors and three LLMs, with additional tests under limited training data and suboptimal feedback. These results provide substantial initial evidence for the method’s effectiveness.
Critical weaknesses
- The results and ablations support TACIT’s design, but I would like to better understand the LLM-only comparison. Both methods get five iterations or attempts and a six-hour time limit, yet the reported median runtime is 2.1 hours for TACIT and 0.2 hours for LLM-only. Do later LLM-only attempts receive solver results or information about what went wrong in earlier attempts? If the baseline could use this feedback to revise its model and run with a similar compute budget, how much of TACIT’s advantage would remain? This would help me judge whether the gains justify the additional computation.
Questions for authors
- Do later LLM-only attempts receive solver results and failure information from earlier candidates? If not, could the authors compare against a baseline that revises its formulation using feedback on the training instances, with similar runtime or API cost?
- The method requires observed target decisions to remain feasible. What happens during initial constraint removal, cut generation, and template completion if a few user corrections violate the true constraints? Is there a way to identify or tolerate such feedback?
3 comments
Response to reviewer qCuf (2/2)
2 - Computational cost and runtime
TACIT indeed requires more computation, primarily because cut search, template completion, and objective calibration themselves solve optimization problems. Importantly, this additional computation provides formal grounding for the formulation repairs through optimization-based parameter fitting and validation against observed feedback, rather than relying on LLM reasoning alone. Our ablations show that optimization-based calibration substantially improves performance over LLM-generated templates alone. But we note that in a production setting, formulation repair could run offline, outside the decision-making hot path, while users continue solving and using the currently deployed formulation. Thus, longer repair runtimes need not translate into longer waits for operational decisions.
Matching computational effort between TACIT and a pure-LLM baseline is not straightforward. For a pure-LLM baseline, natural ways to spend more effort (more independent candidates, longer reasoning, or additional feedback rounds) also increase token usage and API cost. Thus, we do not see a clear way to scale the computational effort of an LLM-only baseline without also increasing its token usage and API cost. We therefore report both runtime and API cost in the tables above and compare against a feedback-enabled LLM baseline at comparable API cost when using GPT-5.6-sol. This comparison does not match wall-clock runtime, but makes the resource tradeoff explicit.
Infeasible target solutions
The user solutions represent the realized solutions that were implemented in the system in the past. As such, the implicit assumption is that since the decisions were implemented, they should be feasible and satisfy all hard constraints. However, these solutions can still be suboptimal depending on the quality of the user corrections, in which case TACIT remains robust as shown in Table B.3.
Thank you to the authors for the detailed information provided in their rebuttal, which has alleviated my concerns to some extent. I appreciate the thorough clarification. Nonetheless, I maintain my original score.
Response to reviewer qCuf (1/2)
Thank you for your positive evaluation and detailed feedback. We address your concern about the LLM-only comparison in two parts: feedback-enabled refinement and computational cost.
1 - Feedback in the LLM-only baseline
The five attempts of the LLM-only baseline reported in the paper are independent, and do not pass information between the attempts. However, this baseline is not naive as it already leverages our separation loss to return the formulation with the lowest separation loss across the five attempts.
Following the reviewer's suggestion, we additionally evaluate two feedback-enabled LLM-only variants over five iterations. At each subsequent iteration, the prompt includes the previous formulation together with its compilation error or the differences between its solutions and the observed targets, in addition to the initial formulation. The first variant returns the last formulation, and the second stops at the first formulation with zero separation loss.
The following tables compare the original baseline, the feedback-enabled baselines, and TACIT. Recall that TACIT uses a small model (always DeepSeek-V4-Flash) for simpler tasks, such as parsing and translating modifications, and a potentially larger backbone model for harder tasks, such as proposing templates and contextualizing cuts. The "small" and "large" token columns refer to these respective roles, with the backbone specified above each table. Each count includes input and output tokens. All figures are per run, and note that the GPT-5.6-sol experiments ran on a slower machine, so runtimes should be compared within each table.
Results with DeepSeek-V4-Flash as the large model.
Results with Kimi-K2.6 as the large model.
Results with GPT-5.6-sol as the large model.
With GPT-5.6-sol, feedback slightly improves the optimality rate from 0.591 to 0.608, while recovery remains at 46.8%. With DeepSeek-V4-Flash and Kimi-K2.6, feedback alone lowers both rates, as the LLM can overwrite a correct formulation in a later revision. Stopping at the first formulation with zero separation loss prevents this and improves over feedback alone with every backbone, raising the optimality rate to 0.303, 0.394, and 0.628 with DeepSeek-V4-Flash, Kimi-K2.6, and GPT-5.6-sol, respectively. TACIT outperforms all LLM-only baselines with every backbone, reaching an optimality rate of 0.757 and a recovery rate of 64.2% with GPT-5.6-sol at a comparable API cost (USD 1.57 per run versus USD 1.31-1.93). Thus, TACIT's advantage persists when the LLM-only baseline can revise its formulation using feedback from previous attempts. With Kimi-K2.6 and GPT-5.6-sol, this advantage also holds at comparable API cost, though not matched runtime.