acceptodds
Under review as a conference paper at ICLR 2027

TACIT: Optimization Models that Learn from Their Mistakes

Abstract

Real-world optimization problems are difficult to model accurately because many objectives and constraints reside in domain experts' tacit knowledge, making them hard to formalize. As a result, optimization models often contain miscalibrated objectives, missing constraints, or omitted decision variables, leading to solutions that fail to reflect operational realities. We address this challenge by automatically repairing misspecified formulations using historical data consisting of past solutions and subsequent user overrides. Traditional approaches such as inverse optimization and constraint learning tend to overfit sparse data and produce complex formulations. Our central idea is to combine the reasoning capabilities and prior knowledge of LLMs with the formal grounding provided by optimization. We realize this idea through two complementary paradigms. Top-down, an LLM proposes structural repairs, including new constraints and variables, whose numerical parameters are calibrated and validated through optimization. Bottom-up, optimization infers cuts from observed decisions, which the LLM contextualizes into interpretable, generalizable modeling constraints. We evaluate our approach on 38 misspecification scenarios spanning nine classes of optimization problems, several drawn from real-world applications, and show that TACIT can repair 78.9% of them (vs. 60.5% for the best baseline).

Primary area:
optimization
Keywords:
Mixed-Integer Programming, Inverse Optimization, Constraint Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qCufOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use decisions from an existing optimization model and user-provided target decisions to correct missing or misspecified constraints, objectives, and variables. TACIT combines two approaches: a bottom-up approach that generates cuts through optimization and uses an LLM to contextualize and generalize them into meaningful constraints, and a top-down approach in which an LLM proposes structural modification templates whose parameters are determined through optimization. Candidate transformations are evaluated using objective calibration and separation loss, and the incumbent formulation is updated iteratively. The authors evaluate TACIT on 38 scenarios spanning nine classes of optimization problems, with target decisions generated by a predefined target model for each scenario. With GPT-5.6-sol, TACIT achieves an average recovery rate of 64.2% across five runs, compared with 46.8% for the LLM-only baseline. The proportion of scenarios recovered in at least one of the five runs is 78.9%, compared with 60.5% for the baseline.

Critical strengths
  1. The problem has practical value. Deployed optimization models often require manual corrections, but these corrections usually appear as changes to decisions rather than explicit explanations of what is wrong with the formulation. The paper uses this feedback to repair the model, including cases that require new auxiliary variables, constraints, and objective terms. This allows it to address structural misspecifications that cannot be resolved by adjusting existing parameters alone.
  2. The recovery criterion is demanding. The repaired model must produce solutions that are feasible under the target model and within 1% of its optimal objective value on all 256 unseen test instances, rather than merely fit the observed corrections. The experiments cover different types of formulation errors and three LLMs, with additional tests under limited training data and suboptimal feedback. These results provide substantial initial evidence for the method’s effectiveness.
Critical weaknesses
  1. The results and ablations support TACIT’s design, but I would like to better understand the LLM-only comparison. Both methods get five iterations or attempts and a six-hour time limit, yet the reported median runtime is 2.1 hours for TACIT and 0.2 hours for LLM-only. Do later LLM-only attempts receive solver results or information about what went wrong in earlier attempts? If the baseline could use this feedback to revise its model and run with a similar compute budget, how much of TACIT’s advantage would remain? This would help me judge whether the gains justify the additional computation.
Questions for authors
  1. Do later LLM-only attempts receive solver results and failure information from earlier candidates? If not, could the authors compare against a baseline that revises its formulation using feedback on the training instances, with similar runtime or API cost?
  2. The method requires observed target decisions to remain feasible. What happens during initial constraint removal, cut generation, and template completion if a few user corrections violate the true constraints? Is there a way to identify or tolerate such feedback?
3 comments
AuthorsOpenReview ↗

Response to reviewer qCuf (1/2)

Thank you for your positive evaluation and detailed feedback. We address your concern about the LLM-only comparison in two parts: feedback-enabled refinement and computational cost.

1 - Feedback in the LLM-only baseline

The five attempts of the LLM-only baseline reported in the paper are independent, and do not pass information between the attempts. However, this baseline is not naive as it already leverages our separation loss to return the formulation with the lowest separation loss across the five attempts.

Following the reviewer's suggestion, we additionally evaluate two feedback-enabled LLM-only variants over five iterations. At each subsequent iteration, the prompt includes the previous formulation together with its compilation error or the differences between its solutions and the observed targets, in addition to the initial formulation. The first variant returns the last formulation, and the second stops at the first formulation with zero separation loss.

The following tables compare the original baseline, the feedback-enabled baselines, and TACIT. Recall that TACIT uses a small model (always DeepSeek-V4-Flash) for simpler tasks, such as parsing and translating modifications, and a potentially larger backbone model for harder tasks, such as proposing templates and contextualizing cuts. The "small" and "large" token columns refer to these respective roles, with the backbone specified above each table. Each count includes input and output tokens. All figures are per run, and note that the GPT-5.6-sol experiments ran on a slower machine, so runtimes should be compared within each table.

Results with DeepSeek-V4-Flash as the large model.

MethodOpt. rateRecovery rateTokens (large)Tokens (small)API cost per run (USD)Mean runtime per run (min)
LLM-only (paper)0.37319.5%174k-0.03211.5
LLM-only with feedback (new)0.21610.5%281k-0.0484.8
LLM-only with feedback + early stop (new)0.30320.5%250k-0.0424.3
TACIT0.60645.3%115k377k0.117148

Results with Kimi-K2.6 as the large model.

MethodOpt. rateRecovery rateTokens (large)Tokens (small)API cost per run (USD)Mean runtime per run (min)
LLM-only (paper)0.46530.0%320k-0.9528.8
LLM-only with feedback (new)0.29816.8%472k-1.2226.4
LLM-only with feedback + early stop (new)0.39428.9%382k-0.9720.8
TACIT0.73455.3%283k363k1.14165

Results with GPT-5.6-sol as the large model.

MethodOpt. rateRecovery rateTokens (large)Tokens (small)API cost per run (USD)Mean runtime per run (min)
LLM-only (paper)0.59146.8%135k-1.3111.3
LLM-only with feedback (new)0.60846.8%228k-1.9320.8
LLM-only with feedback + early stop (new)0.62851.6%156k-1.3113.1
TACIT0.75764.2%68k359k1.57291

With GPT-5.6-sol, feedback slightly improves the optimality rate from 0.591 to 0.608, while recovery remains at 46.8%. With DeepSeek-V4-Flash and Kimi-K2.6, feedback alone lowers both rates, as the LLM can overwrite a correct formulation in a later revision. Stopping at the first formulation with zero separation loss prevents this and improves over feedback alone with every backbone, raising the optimality rate to 0.303, 0.394, and 0.628 with DeepSeek-V4-Flash, Kimi-K2.6, and GPT-5.6-sol, respectively. TACIT outperforms all LLM-only baselines with every backbone, reaching an optimality rate of 0.757 and a recovery rate of 64.2% with GPT-5.6-sol at a comparable API cost (USD 1.57 per run versus USD 1.31-1.93). Thus, TACIT's advantage persists when the LLM-only baseline can revise its formulation using feedback from previous attempts. With Kimi-K2.6 and GPT-5.6-sol, this advantage also holds at comparable API cost, though not matched runtime.

AuthorsOpenReview ↗

Response to reviewer qCuf (2/2)

2 - Computational cost and runtime

TACIT indeed requires more computation, primarily because cut search, template completion, and objective calibration themselves solve optimization problems. Importantly, this additional computation provides formal grounding for the formulation repairs through optimization-based parameter fitting and validation against observed feedback, rather than relying on LLM reasoning alone. Our ablations show that optimization-based calibration substantially improves performance over LLM-generated templates alone. But we note that in a production setting, formulation repair could run offline, outside the decision-making hot path, while users continue solving and using the currently deployed formulation. Thus, longer repair runtimes need not translate into longer waits for operational decisions.

Matching computational effort between TACIT and a pure-LLM baseline is not straightforward. For a pure-LLM baseline, natural ways to spend more effort (more independent candidates, longer reasoning, or additional feedback rounds) also increase token usage and API cost. Thus, we do not see a clear way to scale the computational effort of an LLM-only baseline without also increasing its token usage and API cost. We therefore report both runtime and API cost in the tables above and compare against a feedback-enabled LLM baseline at comparable API cost when using GPT-5.6-sol. This comparison does not match wall-clock runtime, but makes the resource tradeoff explicit.

Infeasible target solutions

The user solutions represent the realized solutions that were implemented in the system in the past. As such, the implicit assumption is that since the decisions were implemented, they should be feasible and satisfy all hard constraints. However, these solutions can still be suboptimal depending on the quality of the user corrections, in which case TACIT remains robust as shown in Table B.3.

Thank you to the authors for the detailed information provided in their rebuttal, which has alleviated my concerns to some extent. I appreciate the thorough clarification. Nonetheless, I maintain my original score.

Reviewer_bs5NOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes TACIT, a framework for refining mixed-integer linear programming formulations using datasets of paired solutions: solutions produced by a misspecified optimization model and corresponding human-corrected solutions reflecting the intended optimization problem. The framework combines three components: First, a bottom-up component, in which an optimization procedure generates new constraints, which are then contextualized and generalized using an LLM. Second, a top-down component, where an LLM proposes modifications to the formulation, potentially including new constraints and auxiliary variables. These modifications are expressed as templates, which are then instantiated with concrete parameter values through an optimization procedure. Third, an objective calibration component, which alters objective coefficients. These components are combined in an iterative search procedure that is guided by a separation loss that encourages the corrected solutions to be preferred over the original, suboptimal solutions.

The paper also introduces a benchmark comprising 38 scenarios across nine classes of linear and mixed-integer linear programs. These scenarios represent different types of formulation errors, including missing constraints, missing variables, and incorrect parameters.

Experiments show that TACIT outperforms both existing data-driven approaches and a direct LLM-based baseline. Ablation studies further demonstrate that all three components contribute to performance, with their combination achieving the best results.

Critical strengths

Principled combination of LLMs and optimization

One of the main strengths of the paper is its principled framework for combining LLMs with optimization methods. I particularly like how the bottom-up and top-down approaches complement each other, with different emphases on the LLM and optimization components. Rather than simply asking an LLM to refine an optimization model, the paper proposes a structured approach in which LLMs and optimization procedures play distinct roles.

I also like the problem setting itself: the method does not merely correct errors in translating a problem description into a formal model, but can address errors in the original understanding of the problem, revealed through user corrections to its solutions. The framework can address a wide range of formulation errors, including missing constraints, auxiliary variables, and objective terms, rather than merely adjusting coefficients.

Extensive experimental evaluation and valuable benchmark

The experimental evaluation is another major strength in my eyes. The comparisons with existing data-driven methods and a direct LLM-only baseline are informative. In particular, Figure 2 clearly illustrates the improvements over direct LLM use, and Table 2 provides a convincing ablation showing that the bottom-up, top-down, and objective calibration components all contribute to performance, with their combination performing best.

I also appreciate the evaluation across different LLM backbones, the appropriate feasibility and optimality metrics, and the experiments on robustness to suboptimal feedback.

Finally, the new benchmark of 38 scenarios across nine LP/MILP problem classes is a valuable additional contribution, covering diverse types of formulation errors. The appendices provide extensive details on the benchmark and methodology, which is always appreciated for reproducability purposes.

Critical weaknesses

Limited assessment of formulation recovery beyond optimal solutions

One weakness I see is that the evaluation focuses only on whether solutions produced by the learned model are feasible and near-optimal under the ground-truth formulation. This does not establish that the learned and ground-truth models have the same sets of optimal solutions, or that they rank suboptimal solutions similarly.

This distinction might matter when optimization is performed heuristically or terminated early, since a near-optimal solution under the learned formulation could be highly suboptimal under the ground-truth model. Evaluating the ground-truth quality of approximately optimal solutions under the learned model would help establish whether the method recovers an appropriate MILP rather than merely reproducing the known optimal decisions.

Synthetic nature of the benchmark The proposed benchmark is synthetic: it starts from known formulations and deliberately introduces errors, rather than using actual human corrections to optimization solutions. Although I understand the practical difficulties of collecting such data, this limits the evidence for how well the method performs on realistic formulation errors and user feedback. I do not consider this a particularly serious weakness, but it is worth noting.

Minor Comments

  • The restriction to LP/MILP formulations is a reasonable scope, but should be stated much earlier, ideally in the abstract or introduction.
  • The acronym AOP (Adapted Optimization Problem) and its meaning could be explained more clearly when first introduced.
  • There appears to be a typo in “the weight vector (to be estimated) for its the family t(k) of this variable.”
Questions for authors
  1. An individual cut may be consistent with several different higher-level constraint families, potentially with similar training separation losses but different generalization behavior. How does the method handle this ambiguity when extrapolating cuts into constraint families?
  2. The evaluation assesses solutions obtained by optimizing the learned formulation, but what happens when the learned formulation is itself solved only approximately? Have the authors considered evaluating the ground-truth quality of solutions that are near-optimal under the learned formulation? This seems particularly relevant when optimization is performed heuristically or terminated early.
  3. The separation loss only requires the observed corrected solution to be optimal, although the ground-truth formulation may admit other optimal solutions. To what extent does the learned formulation preserve these alternative optimal solutions, and could their exclusion affect generalization?
2 comments

Thank you for your supportive review and insightful questions. We address your main concerns and questions below.

Ground-truth quality of solutions that are near-optimal under the learned formulation

Thank you for this suggestion. We manually analyzed the formulations achieving 100% test optimality with GPT-5.6-sol and found that 83% of them are equivalent to the target formulation and thus have the same set of optimal solutions.

To extend the analysis to suboptimal solutions of the learned formulations, we further evaluated the formulations returned by TACIT in our experiments. For every final formulation returned by TACIT, we computed solutions that are restricted to be at least xx\\% worse than the formulation's optimal objective on the 256 test instances, for xx ranging from 5 to 20. We then evaluated these solutions under the target formulation. The table below reports, for recovered formulations (100% test optimality) by TACIT with DeepSeek-V4-Flash and GPT-5.6-sol, the average relative gap under the target formulation of the solutions whose relative gap under the corrected formulation falls in each range.

Relative gap under the corrected formulationDeepSeek-V4-FlashGPT-5.6-sol
5–6% (avg. 5.2%)5.14%5.10%
10–11% (avg. 10.1%)10.04%9.95%
20–22% (avg. 20.1%)19.95%19.78%

For solutions that are 5–6% suboptimal in the corrected formulation, the average absolute difference in relative gap between the ground truth and the corrected formulation is 1.6×10−31.6 \times 10^{-3} for DeepSeek-V4-Flash and 1.8×10−31.8 \times 10^{-3} for GPT-5.6-sol.

Overall, these results indicate that our recovery metric is a good indicator that the corrected formulation matches the target formulation (not merely the target solutions) and that the ground-truth quality of suboptimal solutions closely matches their quality under the corrected formulation.

Synthetic benchmark

The benchmark is indeed synthetic, and we agree that evaluating TACIT on real user overrides is an important direction for future work. Beyond the difficulty of collecting real data, the synthetic data allows us to measure recovery against a known ground-truth formulation (which real user overrides alone do not provide), and cover a variety of misspecification types. That being said, our hand-crafted scenarios are inspired by real applications and cover meaningful use cases that can be described in a single sentence, in contrast to random perturbations of the objective or constraints. We also want to clarify that most of our scenarios do not simply consist of fixing an erroneous formulation of a textbook problem, but rather require uncovering unusual rules or user preferences, such as "a given employee never works two consecutive days". These cannot be inferred from common knowledge and must be learned from the observed corrections.

Minor comments

In addition, we have addressed your "Minor Comments" (stated the MILP/LP scope earlier in the paper, clarified the acronym AOP, and fixed the typo) and taken a general pass verifying clarity and typos.

Generalization behavior

In brief, this is handled by trial and error: the system tries to generalize the cuts into a family, and if the family is detected to be invalid (i.e., any cut in this family excludes an observed target solution), the generalization is rejected. This trial and error behavior is mitigated by the fact that the cut contextualization agent uses accumulated cuts. Thus, as more cuts are generated, it gains additional evidence about how to generalize them. Another contributing factor is that the LLM reasons semantically about the entire formulation, not simply cuts, which supplies an additional inductive bias that helps narrow down the plausible generalization directions.

Alternative optimal solutions

Indeed, our method does not guarantee recovery of the entire set of ground-truth optimal solutions. Regarding preservation of all optimal solutions, as mentioned above, through manual verification we observed that 83% of the recovered instances are equivalent to the target formulation, and thus have the same set of optimal solutions. In particular, while focusing on the feedback provided by a single (among possible multiple) optimal solutions could lead to overfitting, we have not observed generalization issues. Our held-out evaluation supports this, and the small train vs test gaps in feasibility and optimality reported in Table 1 provide empirical evidence against substantial generalization degradation (part of the table reproduced for convenience):

BackboneTrain feasibilityTest feasibilityTrain optimalityTest optimality
DeepSeek-V4-Flash0.790.780.640.61
Kimi-K2.60.870.850.770.73
GPT-5.6-sol0.890.880.790.76

I thank the authors for their clarifications and additional experiments. I would encourage including these additional results in the paper. I am satisfied with the responses to my questions and maintain my positive assessment of the paper and my original rating.

Reviewer_XiStOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors proposed LLM-based repair of optimization models.

The authors considered a scenario where we have data of optimization models and revised solutions done by domain experts. They proposed a LLM-based pipeline that uses the LLM to propose structural repairs, followed by optimization to determine numerical parameters, and to interpret and revise cuts determined from optimization. This division of tasks sounds reasonable to me.

The authors demonstrated the performance gain over data-driven methods and LLM-only baselines in a variety of scenarios.

Critical strengths

Novelty. This paper is one of the first to utilize LLMs for optimization model repair. Although LLMs have been increasingly used for formulating optimization models, model repair is a different task and presents a distinct set of challenges. Addressing these challenges is a novel contribution.

Critical weaknesses

Motivation of Problem Formulation. I think more motivation can be given in terms of the problem formulation.

  • The authors assumed that the domain expert provide the revised solutions, but not explanations of why or what revisions are made. It seems to me that it is not hard to ask the domain expert to write some explanations during the review process (e.g., "we need motors to assemble the machine" in your toy example in Section 1). Under these reasonable scenarios, the need for a method that "guess" the model revision from the revised solution is reduced.
  • Another implicit assumption is that we already knew the problem formulation is wrong. I think in practice, we would need to determine whether the formulation is wrong first, and then repair if necessary.
Questions for authors

Some responses to my comments in Critical Weaknesses would be helpful.

Other questions are as follows.

  • Are you able to extend the framework to a general nonlinear (but maybe convex) optimization problem? If not, why? It would be good to state explicitly that the paper deals with LPs and MILPs. Right now, this assumption is hidden in Line 167.
  • If I understand correctly, the method is not zero-shot, because there are 128 training instances and 256 test instances. Could you explain what exactly are trained?
  • Why do AOP linear and AOP tree perform very poorly in Table 1 (e.g., 0% recovery). I assume their methods have non-zero recovery rates in their papers... Can you give some explanations of these results?
  • Can you compare the complexity of the proposed method and the LLM-only baselines, especially in terms of tokens? Can the LLM-only baselines call tools, perform reasoning, and have multiple rounds of revisions?
4 comments
AuthorsOpenReview ↗

Response to reviewer XiSt (1/3)

Thank you for your positive evaluation and constructive feedback. Below, we address your points.

Problem setting and assumptions

  • Thank you for this question. While explanations are certainly valuable when available, requiring them may impose an additional burden on users, whereas corrected decisions are already recorded as part of routine operations. Practitioners may know how to adjust a recommendation without being able to articulate the general modeling rule behind that adjustment, especially when it reflects tacit knowledge. This motivation also reflects our experience deploying optimization systems in production, where users are often reluctant to provide feedback beyond the minimum required for their operational tasks. Nevertheless, if such explanations happen to be available, our framework can easily leverage them by incorporating them into the prompt used to generate templates and contextualize cuts. In a preliminary experiment with DeepSeek-V4-Flash on the 10 hardest scenarios, adding short user explanations to the prompts yielded comparable gains for TACIT and the LLM-only baseline (optimality rate improved from 0.208 to 0.275 and from 0.168 to 0.242, respectively), mainly driven by specific explanations (e.g., "capacity requirement was violated"). TACIT remains valuable in this setting, since even if users know the general nature of the required change, they may not know its exact numerical parameters, which is precisely what the objective calibration and constraint completion modules identify.
  • We agree that detection should precede repair, and in our setting it is provided by the user feedback itself. We assume that users always accept the solution if the formulation is correct, which we believe is a reasonable working assumption: if they change it, there should be at least one constraint or preference that is not currently captured. Under this assumption, user corrections implicitly reveal incorrect formulations, and our system is triggered only when such corrections are observed. Conversely, if users accept the recommended solutions on every instance they encounter, the formulation is aligned with their intent on the instances encountered and can, for operational purposes, be regarded as correct.

Extension to nonlinear problems

We indeed consider only MILPs, with LPs as a special case, and we have made this clear earlier in the revised version. Although our implementation handles only MILPs, the framework extends to nonlinear (convex or nonconvex) problems with templates that allow nonlinearity in the decision variables, provided the templates remain linear in the learnable parameters. Each generated constraint evaluates the template at fixed solutions, so objective and constraint calibration are unchanged and their master problems remains LP and a MILP, respectively. Only the class of subproblems changes: they inherit the nonlinearity of the templates, becoming convex or nonconvex.

What is trained

We thank the reviewer for the opportunity to clarify this distinction. While TACIT uses historical observations to repair a formulation, it does not train or fine-tune the LLMs. For each formulation to be repaired, our system receives kk (=128=128 in our main experiments) instances of this formulation with observed solution corrections (e.g., from use of the formulation on the previous days). Note that these are not examples of other repaired formulations. These observations guide the structural search and the optimization-based fitting of numerical parameters in the repaired formulation. The resulting formulation is then evaluated on 256 held-out instances. Thus, "training" refers to correcting the optimization formulation using past user-corrected solutions, not training the LLM or learning a repair procedure from other formulation repairs.

AuthorsOpenReview ↗

Response to reviewer XiSt (2/3)

AOP performance

Our recovery and recoverability metrics impose a substantially stronger criterion than the metrics reported in these works: a scenario is recovered only if the repaired formulation produces feasible solutions within 1% of ground-truth optimality on all 256 held-out instances. Recoverability requires at least one of five runs to meet this same criterion. Bayani et al. instead report feasibility rates and average objective gaps. Consequently, their positive results do not necessarily imply recovery under our definition. For example, in their Table 3, the best-feasibility methods achieve 100% feasibility but average gaps of 3% and 6% on linear-knapsack Set4 and Set5, respectively. Neither scenario would count as recovered under our criterion. Similarly, their classification-tree method achieves 100% feasibility but average gaps of 19% and 12% on binary-knapsack Set3 and Set6 (Table 4).

These results reflect a feasibility-optimality tradeoff: increasing the penalty coefficient cc in the AOP methods encourages consistency with the learned correction mapping, but can sacrifice objective quality. Bayani et al. explicitly discuss this behavior in Section 4.1.2 and Figure 2, and our Table B.4 exhibits the same qualitative pattern: increasing cc from 1 to 256 raises AOP-linear test feasibility from 51% to 62%, while reducing the fraction of jointly feasible, near-optimal solutions from 13% to 1%. AOP-tree exhibits a similar tradeoff.

Finally, effectiveness varies across scenarios and depends on the nature of the formulation misspecification. Bayani et al. obtain excellent classification-tree results on several nurse-rostering settings, but substantially larger objective gaps on knapsack settings. Our benchmark covers different problems and a broader range of misspecifications: their experiments only consider missing constraints, while most of the scenarios of our benchmark also involve incorrect objectives, missing variables, and incorrect variable domains. The 0% recovery in our setting, therefore, does not contradict their reported successes. It reflects performance on this broader benchmark under a stringent joint feasibility-and-optimality criterion.

AuthorsOpenReview ↗

Response to reviewer XiSt (3/3)

Token usage and capabilities of LLM-only baselines

All LLMs use high reasoning effort. The LLM-only baselines are not merely chat-based: they generate executable gurobipy code, which the evaluation pipeline executes and solves using Gurobi. Thus, they use code execution and a solver through the surrounding pipeline, although the LLM does not autonomously invoke tools.

In the original baseline, five attempts are run independently (without feedback) and the formulation with the lowest training separation loss is returned. However, we have now evaluated two additional LLM-only baselines with feedback: each subsequent prompt includes the previous formulation and its compilation error or the differences between its solver-produced solutions and the observed targets, together with the initial formulation. After five iterations, the final formulation is returned. The second variant stops at the first formulation with zero separation loss.

The table below reports token usage, API cost, and runtime per run of these LLM-only baselines and our system, TACIT. Recall that TACIT uses a small model (always DeepSeek-V4-Flash) for simpler tasks, such as parsing and translating modifications, and a potentially larger backbone model for harder tasks, such as proposing templates and contextualizing cuts. The "small" and "large" token columns refer to these respective roles, with the backbone specified in the first column. Each count includes input and output tokens.

BackboneMethodTokens (large)Tokens (small)API cost (USD)Mean runtime (min)
DeepSeek-V4-FlashLLM-only (paper)174k-0.03211.5
DeepSeek-V4-FlashLLM-only with feedback (new)281k-0.0484.8
DeepSeek-V4-FlashLLM-only with feedback + early stop (new)250k-0.0424.3
DeepSeek-V4-FlashTACIT115k377k0.117148
Kimi-K2.6LLM-only (paper)320k-0.9528.8
Kimi-K2.6LLM-only with feedback (new)472k-1.2226.4
Kimi-K2.6LLM-only with feedback + early stop (new)382k-0.9720.8
Kimi-K2.6TACIT283k363k1.14165
GPT-5.6-solLLM-only (paper)135k-1.3111.3
GPT-5.6-solLLM-only with feedback (new)228k-1.9320.8
GPT-5.6-solLLM-only with feedback + early stop (new)156k-1.3113.1
GPT-5.6-solTACIT68k359k1.57291

TACIT uses fewer backbone-model tokens than any LLM-only baseline. Although it uses more tokens overall, the additional tokens are processed by the smaller model. Consequently, with Kimi-K2.6 and GPT-5.6-sol backbones, TACIT’s total API cost remains comparable to the LLM-only baselines (USD 1.14 versus USD 0.95-1.22, and USD 1.57 versus USD 1.31-1.93 per run). With every backbone, TACIT also achieves higher optimality and recovery rates than all three LLM-only variants (see the tables in our response to Reviewer qCuf).

Regarding runtime, TACIT's longer runtime primarily reflects the optimization problems solved for cut search, template completion, and objective calibration. Importantly, this computation provides formal grounding for the formulation repairs and contributes substantially to their quality: Table 2 shows that adding optimization-based template completion raises recovery from 21.1% to 41.1%. Moreover, formulation repair could run offline, outside the decision-making hot path, while users continue using the currently deployed formulation. Longer repair runtimes therefore need not translate into longer waits for operational decisions. (Runtimes should be compared within each backbone, since they were run on different machines.)

I thank the authors for their responses, which clears the confusion on nonlinear programming, the training, and the performance of the AOP baselines. The authors also demonstrated the low cost of the proposed method in terms of token usage.

I am not fully convinced about the argument on detecting the incorrect formulation. So the premise is basically that the user will detect incorrect models. I think if the user can detect it, the issues to repair are probably clear. But issues that are hard to repair are also hard to detect.

I will maintain my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.