acceptodds
Under review as a conference paper at ICLR 2027

COOPA: A Modular LLM Agent Architecture for Operations Research Problems

Abstract

Operations Research (OR) provides a rigorous framework for high-stakes decision-making, but effective OR modeling requires substantial domain knowledge, mathematical abstraction, and solver expertise. Recent LLM-based systems automate parts of this pipeline, yet remain limited by low accuracy on complex problems, opaque outputs, and narrow solver support. We propose COOPA (**CO**operative **OP**erations **A**gent), a modular LLM-agent architecture for interpretable and scalable OR decision support. It combines three components: iterative confidence-based modeling, which generates multiple candidate formulations, self-evaluates them across modeling dimensions, and selects one using a max-min confidence criterion; element-level provenance and confidence explanations, which link variables, parameters, constraints, and objectives to quoted source text and provide an audit trail for human verification; and multi-solver routing to specialized optimizer agents for different OR problem classes. Across three OR benchmarks, seven LLM backbones, and four baselines under identical conditions, COOPA achieves the best macro-average accuracy on six of seven backbones and improves over the strongest baseline by up to 6.7 percentage points. A within-system ablation evaluates the contribution of iterative confidence-based modeling, while additional quantitative analyses validate source traceability and multi-solver dispatch.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model Agents, Operations Research, Optimization Modeling, Mathematical Programming

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ofnfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

COOPA is a modular optimization-modeling agent with three components: schema-validated formulations linked to quoted source text; generation and evaluation of multiple history-informed candidate formulations; and dispatch to specialized solver agents. Its default selection rule chooses the candidate with the highest minimum self-assessed confidence across parameters, variables, objectives, and constraints. Evaluation on three benchmarks and eight backbones reports 64.8% average accuracy, versus 61.6% for the strongest baseline by that aggregate measure, with COOPA leading on six backbones. Iteration, repeated-run, and routing studies supplement the main comparison. Accuracy is measured through agreement with reference objective values.

Critical strengths
  • The architecture explicitly addresses executable but semantically incorrect formulations. Formulation assessment occurs before solver-code generation, rather than relying exclusively on execution failures to trigger correction. Structured modeling elements and source quotations also make individual modeling choices inspectable. The running example illustrates the intended benefit: an executable sum-of-efficiencies objective is revised into the appropriate bottleneck-throughput objective. This is a relevant failure mode that ordinary debugging alone would not necessarily catch.
  • There is meaningful empirical support beyond a single backbone or demonstration. The paper compares eight backbones, repeats selected comparisons three times, and studies refinement-budget sensitivity. It also reports a routing comparison on 164 DCP-Bench instances, where the full system improves over always using the mathematical optimizer by 8.74 percentage points, with a positive problem-clustered confidence interval. These checks strengthen the evidence that parts of the architecture are useful, even though the benefits are not universal.
Critical weaknesses
  • The experiments do not isolate the value of max-min confidence selection from additional refinement and inference expenditure. The principal ablation compares the first formulation against selection from three history-refined formulations. That combines better candidate generation with candidate selection; it does not establish that max-min outperforms simply taking the last candidate, random selection from the same pool, or another selector. The confidence–accuracy correlation does not resolve this confounding. This matters because the cost study finds approximately seven times the API cost of OR-LLM-Agent on the sampled Gemini-2.5-Flash workloads, while the overall accuracy advantage is modest.
  • The claimed human-verification benefit remains an architectural proposition rather than a validated outcome. The paper does not measure whether provenance and confidence explanations improve human error detection or reduce verification time, and it does not separately audit the accuracy and completeness of source references. Its confidence analysis establishes some association with improvement, but not calibrated probabilities of correctness. Consequently, the outputs may be useful inspection aids, but the evidence does not yet establish that they make human reliance on the generated models more reliable.
Questions for authors
  • Does COOPA genuinely detect semantic formulation errors, or does it mainly benefit from additional sampling and refinement? The current ablation compares the full k=3k=3 procedure against using only the first candidate, which conflates additional candidate generation, self-refinement, and confidence-based selection. Could the authors compare against selecting the last candidate, random selection, or best-of-kk under alternative selectors using exactly the same candidate pool and inference budget?
  • How reliable is the confidence evaluator specifically at identifying semantic errors? Could the authors evaluate whether low confidence correctly localizes the erroneous component (parameter, variable, objective, or constraint) using annotated formulation errors independently? The current end-to-end accuracy results do not establish whether the evaluator itself can reliably detect semantic mistakes.
  • What happens when the model is confidently wrong? Since both formulation generation and correctness assessment are performed by LLMs without an external semantic verifier, can the authors quantify cases where an incorrect formulation receives uniformly high confidence and is therefore not repaired?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ofnf [1/1]

Thank you for your careful review and constructive suggestions. We have added comparisons of selection rules on the same candidate pools [CQ1] and performance at comparable API costs [CQ2], together with a source-reference audit [CQ3], to address your concerns. We also respond below to your questions about verification and confidence reliability.

[Q1] Do the gains come from refinement, final selection, or additional computation?

Following your suggestion, the new comparison in [CQ1] holds the generated candidate pools and refinement process fixed while changing only the final selection rule. Max-min therefore receives exactly the same candidates as Last Candidate, Random Selection, and max-mean. It achieves higher macro-average success rates than Last Candidate and Random Selection on both backbones, including gains of 1.20 percentage points over Last Candidate on GPT-5.2 and 0.78 points on Gemini-3-Flash. Max-mean achieves similar results. These direct comparisons support the value of confidence-guided final selection on an unchanged refinement history, with the benefit observed under both minimum and mean aggregation.

To test whether additional baseline attempts close the performance gap at a similar cost, [CQ2] compares all four baselines using GPT-6.1-Sol and reports their actual API costs. For example, OR-LLM-Agent receives four independent attempts combined by majority voting, achieving 65.66% macro-average success at $0.145 per problem. COOPA achieves 73.55% at $0.148 with one complete pipeline run, including its internal refinement. COOPA also retains the highest macro-average success rate across the four baseline comparisons at comparable costs. This provides evidence that its advantage persists when competing methods receive a similar API budget for additional attempts.

[Q2] Do source references and confidence explanations improve human verification, and how are the references themselves evaluated?

Source references connect individual modeling elements to the problem description, allowing a formulation to be inspected component by component. For example, an interface could display each parameter, variable, objective, and constraint alongside its cited passage and the corresponding confidence explanation. For example, a reviewer could check whether a capacity constraint uses the limit stated in the cited passage, with the evaluator’s explanation available to help examine the modeling choice. Evaluating the effects of these features on human error detection and verification time requires dedicated user studies, which are beyond the scope of this paper, as discussed in Appendix F.2.

The new audit in [CQ3] provides quantitative evidence on the reliability of these source links, with most references judged to support their associated source attributions. The audit evaluates the references supplied with the generated elements; whether they cover all source information needed to verify the full formulation would require a separate completeness assessment.

COOPA uses the evaluator's scores to guide refinement and rank candidate formulations for the same problem. These scores are not calibrated probabilities of correctness; for example, a score of 90 does not imply a 90% probability of correctness. The same-pool comparison in [CQ1] directly evaluates their use for final selection and supports their value in choosing among candidate formulations.

[Q3] Do low confidence scores identify incorrect components, and how often does an incorrect formulation receive uniformly high confidence and remain uncorrected?

The three main benchmark datasets used in our evaluation provide reference optimal objective values but no reference mathematical formulations. We therefore follow the benchmark protocol and check whether the generated model produces the reference optimum. This is a useful indicator of successful modeling: errors in the variables, objective, or constraints can change the optimal value, so matching the reference value tests whether the model reproduces the expected optimization result.

The new comparison in [CQ1] further tests whether confidence helps select formulations that achieve this result. With candidate pools held fixed, both max-min and max-mean achieve higher macro-average success rates than Last Candidate and Random Selection on both backbones. These gains provide evidence that the evaluator's assessments contain useful information for selecting successful candidates.

Quantifying how accurately low confidence identifies erroneous components, and how often semantic errors receive uniformly high scores and persist through refinement, would require an additional audit with independently annotated formulations across refinement rounds. Such an audit would build on the selection evidence in [CQ1] to assess how consistently the evaluator identifies specific modeling errors and how often subsequent revisions correct them.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer ofnf,

Thank you again for your detailed and constructive comments. To address your concerns about refinement, selection, and inference cost, we added comparisons that hold the candidate pools fixed while changing the final selection rule, together with comparisons at similar API costs. We also audited source-reference reliability and clarified how the references support component-by-component inspection, how confidence scores are used, and the scope of the evidence supporting these uses.

We would appreciate hearing whether these results and explanations address your concerns, and whether there are remaining points we could clarify during the discussion.

Thank you for the selection ablations, cost-matched comparisons, and source-reference audit. These strengthen the evidence for the overall workflow. However, max-min shows no clear advantage over max-mean, and the audit establishes source support rather than formulation correctness. I maintain my rating of 3: the additional evidence is encouraging, but the methodological contribution remains modest for a stronger recommendation.

AuthorsOpenReview ↗

Response to Reviewer ofnf

Dear Reviewer ofnf,

Thank you for carefully considering our additional experiments and for your constructive feedback throughout the discussion. We respect your assessment and your decision to maintain your rating. We would like to briefly clarify the two points you raised, in case our earlier explanations were not sufficiently clear.

First, max-min is a reasonable default, motivated by the fact that a single incorrect modeling component can invalidate an entire formulation. We do not claim that it should outperform every alternative. Both max-min and max-mean achieve higher average success rates than Last Candidate and Random Selection, supporting the effectiveness of our confidence-guided selection approach across different aggregation rules.

Second, as explained in [Q3], the benchmarks provide only reference objective values, not ground-truth formulations. An LLM tasked with auditing formulation correctness must therefore independently resolve the same semantic modeling questions posed by the natural-language problem description. Our GPT-6.1-Sol experiments demonstrate that even frontier models continue to make errors on these tasks, meaning that another LLM's judgment cannot automatically be treated as reliable ground truth. This is fundamentally different from source-reference auditing, where an LLM checks whether the cited text supports the attribution of an individual modeling element, such as a variable or objective.

More importantly, agreement with the reference optimum is already a strong indicator of formulation correctness under the existing benchmark protocols. Most benchmark problems are specified precisely enough to admit a single intended mathematical interpretation. In these cases, reproducing the reference optimum provides strong evidence that the generated formulation captures the intended problem, even without directly comparing formulations. For the relatively few problems whose descriptions admit multiple valid interpretations and, consequently, multiple defensible formulations, there may be no uniquely defined ground-truth formulation against which to assess correctness. In such cases, requiring agreement with a particular formulation would risk rejecting valid alternatives, whereas objective-value comparison provides a consistent and operationally well-defined evaluation criterion under the established benchmark protocols. Establishing reliable formulation-level correctness labels beyond these existing evaluations would therefore require substantial additional annotation, explicit treatment of ambiguity, and independent validation.

Reviewer_b7tmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COOPA, a modular LLM-agent framework for operations research problems. It combines structured mathematical modeling, iterative confidence-based refinement, and multi-solver routing. Experiments across three benchmarks and eight LLM backbones demonstrate improvements over existing agent-based methods

Critical strengths

The paper addresses a practical challenge in LLM-based optimization and presents a clear modular architecture. The evaluation across multiple benchmarks and LLM backbones is relatively comprehensive.

Critical weaknesses
  1. Limited novelty and insufficient ablations. The core method closely resembles Self-Refine and Best-of-N strategies, while the max-min selection rule lacks sufficient justification. The ablations cannot distinguish refinement gains from selection gains. Comparisons with Random Selection, Last Candidate, Max-Mean, and standard Self-Refine are missing, and the reported confidence statistics cannot replace these comparisons.

  2. High cost and insufficient validation. COOPA improves accuracy from 61.6% to 64.8% but incurs approximately 7× the API cost, without compute-matched comparisons. Moreover, Source Traceability lacks quantitative evaluation, while Multi-Solver Routing is insufficiently validated, with 91.1% of main-benchmark calls using the mathematical optimizer and limited additional coverage.

Questions for authors
  1. What is the specific advantage of max-min confidence selection over simply using the last refined candidate? How much of the improvement can be attributed to the selection mechanism itself?
  1. Given the substantial increase in inference cost, would COOPA still outperform simpler refinement or sampling strategies under the same computational budget?
5 comments
AuthorsOpenReview ↗

Response to Reviewer b7tm [1/2]

Thank you for your careful review and constructive suggestions. We have added experiments on candidate selection, API cost, and source-reference reliability to address your concerns, as reported in [CQ1–CQ3] of our common response. We also clarify how COOPA builds on self-refinement and the practical value of its multi-optimizer design below.

[Q1] How does COOPA relate to Self-Refine and Best-of-N, and what do comparisons with alternative selection rules show?

COOPA's contribution is a formulation-level workflow tailored to optimization modeling. It explicitly represents parameters, variables, objectives, and constraints, links them to source text, and evaluates and refines them before passing the selected formulation to a suitable optimizer agent.

COOPA's iterative modeling process follows the feedback-and-revision idea of Self-Refine: each new formulation uses earlier candidates and their assessments (Section 3.3, Algorithm 1). Last Candidate returns the final formulation from this process. Best-of-N refers to selecting a candidate from several alternatives; the max-min and max-mean rules apply this selection idea to COOPA's history of refined formulations using component confidence scores.

Max-min is a simple default rule within this workflow, motivated by the fact that one incorrect modeling component can invalidate an otherwise sound formulation. High confidence in other components should therefore not offset a low score on that component.

Following your suggestion, [CQ1] compares max-min, max-mean, Last Candidate, and Random Selection on exactly the same candidate pools, with First Candidate as a reference. This separates the effect of refinement (First Candidate versus Last Candidate) from the added contribution of final selection (Last Candidate versus max-min or max-mean).

The effect of refinement alone varies across backbones. Last Candidate improves the macro-average from 67.90% to 68.87% on Gemini-3-Flash, while on GPT-5.2 it scores 69.02%, compared with 69.74% for First Candidate. Max-min's macro-average exceeds Last Candidate by 1.20 percentage points on GPT-5.2 and 0.78 points on Gemini-3-Flash.

These results highlight the value of COOPA's refinement-and-selection workflow: both confidence-based rules achieve higher macro-average success rates than First Candidate, Last Candidate, and Random Selection on both backbones. In particular, the gains over Last Candidate, with refinement held fixed, support the additional value of selecting from the candidate history. The similar results for max-min and max-mean further show that COOPA can benefit from different confidence aggregation rules.

[Q2] Does COOPA retain its accuracy advantage when the baselines receive a similar API budget?

Following your suggestion, we added the experiments in [CQ2] to compare COOPA with all four baselines under similar API budgets. Using GPT-6.1-Sol throughout, we give OptiTree (Fixed), Chain-of-Experts, and OR-LLM-Agent multiple independent attempts and combine their reported objective values by majority voting. We vary the attempt count for each method, including configurations with costs close to COOPA's, and report the actual API cost per problem. OptiMUS is included with one run, whose cost already exceeds COOPA's.

The results show that additional attempts bring limited gains, while COOPA achieves the highest macro-average success rate at comparable API costs. This supports the effectiveness of COOPA's workflow when competing methods receive more attempts and extends the evidence to the newer GPT-6.1-Sol backbone.

AuthorsOpenReview ↗

Response to Reviewer b7tm [2/2]

[Q3] How are source traceability and multi-solver routing validated, given that most main-benchmark calls use the mathematical optimizer?

Following your suggestion, we added the source-reference audit in [CQ3]. It checks whether source quotes are substrings of the original problem and whether the cited information supports the associated modeling choices. Most references were judged to provide this support, giving quantitative evidence on the reliability of COOPA's source links.

For multi-solver dispatch, the main advantage is flexibility across applications. COOPA automatically selects an optimizer based on the formulation, and developers can add or replace optimizer agents without changing the modeling stage. This reduces the need for manual solver configuration, particularly for users unfamiliar with the available optimization methods.

Our experiments illustrate how the same framework uses different optimizers for different problem sets. On the main benchmarks, the mathematical optimizer accounts for 91.1% of optimizer calls (Appendix D.5, Table 8). On the 164 discrete combinatorial problems in DCP-Bench, COOPA selects the combinatorial optimizer for 92.3% of run–problem pairs (Section 4.4).

The DCP-Bench study also compares against the same system restricted to always using the mathematical optimizer. This control reflects the restriction to mathematical programming that is common in existing OR agents. Allowing COOPA to select among optimizer agents improves mean accuracy by 8.74 percentage points over this restricted configuration, demonstrating the practical benefit of adapting the optimizer to the problem.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer b7tm,

Thank you again for your careful review and helpful suggestions. Our rebuttal clarifies COOPA’s relationship to Self-Refine and Best-of-N and reports new comparisons that distinguish refinement gains from the contribution of final selection. We also added comparisons at similar API costs and a source-reference audit, and explained the practical flexibility provided by the multi-optimizer design.

We would appreciate your assessment of these additions and whether they address your concerns about the contribution and supporting evidence. If any aspect remains unclear, we would be happy to provide further clarification.

Reviewer_aHKLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that when LLMs perform operations research modeling, their main errors occur in the modeling stage, while most existing methods generate a model only once and rely on execution errors for correction. To address this, the authors propose COOPA: convert the problem statement into a model with source quotes attached, have an LLM score and explain it along four dimensions—parameters, variables, objective, and constraints—then regenerate with reference to existing candidates and scores, producing 3 candidates in total; finally, select one according to the max-min criterion and route it to Pyomo, OR-Tools, pymoo, or a Python optimization agent for solving.

On 3 benchmarks and 8 backbones, COOPA achieves an average cross-model accuracy of 64.8% (the strongest baseline is 61.6%) and is best on 6 of 8 backbones, but notably lags behind on Qwen3-30B. In a within-system ablation, the full k = 3 pipeline improves on using only the first candidate by 3.0 points. COOPA's per-problem cost is about 7x that of OR-LLM-Agent.

Critical strengths

1. The problem is well-motivated. LLM-generated OR formulations can execute without errors even when the underlying math is wrong. Because execution feedback cannot catch such errors, checking the formulation before code generation is a sensible direction. The assembly-line example illustrates this clearly: the first candidate runs without issue, but maximizes the sum of efficiencies rather than the bottleneck throughput.

2. The evaluation is broad and transparently reported. The paper compares methods across eight backbones and three benchmarks, repeats runs three times on two backbones, and openly reports the failure cases on Qwen3-30B in the appendix. The observation that the strongest baseline varies across backbones also shows why cross-model evaluation is necessary.

3. The intermediate artifacts are easy to inspect. Attaching source references to each modeling element and providing short explanations for the four dimension scores gives users a concrete starting point for manual verification, which could reduce the need to re-read the full problem description when checking the model.

Critical weaknesses

1. The specific contribution of confidence-based selection remains unclear. Table 7 compares the first candidate against the candidate selected from three candidates, the latter two generated with earlier candidates and scores as context. Because later candidates already incorporate feedback from earlier rounds, the gains may come from iterative revision itself or from max-min selection, the latter being the paper's core mechanism. The 181 beneficial versus 95 harmful overrides show that the overall loop is useful, but they do not show that max-min outperforms simply taking the last candidate. Comparing max-min, last-candidate, and random selection on the same set of candidates would clarify how much the confidence scoring actually contributes.

2. Part of the gain may come from additional compute. COOPA costs roughly 7x as much per problem as OR-LLM-Agent, whereas the baselines do not compare multiple candidate formulations before code generation. It remains unclear how much of the 3.2-point average advantage would remain if strong baselines were given a matched budget for multiple attempts or simple self-revision.

3. Neither formulation correctness nor source-reference reliability is directly verified. The evaluation metric only checks whether the objective value falls within 0.1 of the reference optimum. Matching the objective value does not guarantee that the variables and constraints are formulated correctly, even though the abstract claims improvements in formulation accuracy. A similar issue applies to the source references: Appendix F.2 acknowledges that their accuracy and completeness are not audited. Addressing this does not require a large-scale user study; auditing the formulations and references on a sample of problems where iterations changed the outcome would substantially strengthen these claims.

Questions for authors

1. Role of the selector. Using the existing three-round candidates, could you compare the accuracy of max-min selection, taking the last candidate, and random selection on the same candidate pool?

2. Compute overhead. Within COOPA, how many additional tokens and how much extra cost does k = 3 incur compared to k = 1? The roughly 7x figure cited in Appendix F.3 comes from the cross-system comparison against OR-LLM-Agent in Appendix D.6, and therefore does not directly measure the overhead of iteration itself. If feasible, please also report budget-matched results for a strong baseline (e.g., best-of-3 or majority voting) on one or two backbones.

3. Correctness of formulations and source references. Could you audit a sample of problems where iteration changed the outcome (e.g., where the first candidate failed but the selected candidate succeeded) to verify whether their variables, constraints, and source references are actually correct?

4. Routing experiment. On DCP-Bench, 92.3% of run–problem pairs are routed to the combinatorial optimizer. Could you report the accuracy of an "always use the combinatorial optimizer" baseline to separate the benefit of adaptive dispatch from simply switching the solver backend? Please also clarify the source, selection criteria, and evaluation metric for these 164 instances.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aHKL [1/2]

Thank you for your careful review and constructive suggestions. We have added comparisons of candidate selection rules [CQ1 and Appendix D.7] and performance at similar API costs [CQ2 and Appendix D.8], together with a source-reference audit [CQ3 and Appendix D.9]. We also report the cost of refinement within COOPA [Q4 and Appendix D.6] and clarify the DCP-Bench dataset and evaluation protocol [Q5 and Appendix D.5].

[Q1] What does confidence-based selection contribute, and how does max-min compare with other selection rules on the same candidates?

We use max-min as a simple selection rule within COOPA’s formulation-level workflow. Its motivation is that a single incorrect modeling component can invalidate an otherwise sound formulation, so a low confidence score on that component should not be offset by high scores elsewhere.

Following your suggestion, we compared max-min, max-mean, Last Candidate, and Random Selection on exactly the same candidate pools, as reported in [CQ1] of our common response. Both confidence-based rules achieve higher macro-average success rates than Last Candidate and Random Selection on both backbones. With candidate generation held fixed, these results support the value of confidence-guided final selection in COOPA, beyond returning the final self-refined formulation. This advantage is observed with both minimum and mean aggregation, supporting max-min as a simple, effective default while showing that COOPA can also benefit from alternative confidence-based selection rules.

[Q2] Would strong baselines match COOPA if given a similar API budget for multiple attempts or self-revision?

The additional experiments in [CQ2] show that COOPA retains its advantage when the baselines are given more attempts at comparable API costs. We evaluate all four baselines. For OptiTree (Fixed), Chain-of-Experts, and OR-LLM-Agent, we increase the number of independent runs and combine their outputs by majority voting. For OptiMUS, we report a single run, which already costs more per problem than COOPA. The gains from additional attempts are limited, and COOPA achieves the highest macro-average success rate among the evaluated configurations. This supports the effectiveness of COOPA’s workflow under a comparable spending budget. Using GPT-6.1-Sol throughout this comparison also extends the evidence to a newer backbone.

[Q3] Can formulation correctness and source-reference reliability be checked directly, including on problems where refinement changes the outcome?

The benchmark datasets used in our evaluation provide reference optimal objective values but no reference mathematical formulations. We therefore follow the benchmark protocol and check whether the generated model produces the reference optimum. This is a useful indicator of successful modeling: errors in the variables, objective, or constraints can change the optimal value, so matching the reference value tests whether the model reproduces the expected optimization result.

To directly assess source-reference reliability, we added the audit reported in [CQ3]. It examines references in COOPA’s selected formulations across all three benchmarks, checking whether the quoted text appears in the original problem and whether it supports the associated modeling element. Most references were judged to provide this support. These results provide quantitative evidence that COOPA’s source references connect individual modeling choices to supporting information in the original problem description.

AuthorsOpenReview ↗

Response to Reviewer aHKL [2/2]

[Q4] How many additional tokens and how much extra cost does k=3 incur compared with k=1? The sevenfold figure compares COOPA with OR-LLM-Agent rather than measuring refinement within COOPA.

Following your suggestion, we directly measure the token usage and API cost of COOPA with k=1 and k=3 using Gemini-2.5-Flash.

Refinement budgetAvg. input tokens / problemAvg. output tokens / problemAvg. total tokens / problemAvg. cost / problem
k=1150,50118,568169,069$0.0596
k=3133,23236,563169,795$0.1050

Absolute costs differ from Appendix D.6, Table 9 because of difference in problem samples; token accounting and cost calculation are unchanged.

Increasing k from 1 to 3 adds only 726 total tokens per problem (+0.4%) and $0.0454 in API cost. The average cost of k=3 is $0.1050 per problem, approximately 1.8 times that of k=1.

Output tokens increase by 17,995, while input tokens decrease by 17,269, leaving total token usage nearly unchanged. The higher API cost reflects the increased output usage, since output tokens have a higher unit price. All token counts are summed across the complete pipeline, including repeated context across LLM calls, and the cost calculation includes cached-input discounts.

[Q5] How does routing compare with always using the combinatorial optimizer, and what are the source, selection criteria, and metric for the 164 DCP instances?

The main benefit of supporting multiple optimizer agents is practical flexibility. Optimizers can be added or replaced to suit an application without changing the modeling stage (see Section 3.4 and Appendix F.1). COOPA chooses an optimizer automatically based on the formulation, so users do not need to select a solver manually. This is particularly useful for users who are unfamiliar with the available optimization methods.

This flexibility does not imply higher accuracy on every benchmark. DCP-Bench focuses on combinatorial problems, for which fixing the system to use the combinatorial optimizer is the correct choice. We do not expect routing to outperform that fixed choice on this benchmark.

Our evaluation uses the full 164-problem DCP-Bench-Open dataset [1]. The benchmark authors selected problems with available CPMpy reference models to enable automated evaluation. We evaluate all 164 problems without selecting a subset. We follow the benchmark’s solution-level metric: a returned solution must satisfy all constraints in the reference model and, for an optimization problem, attain the reference optimal objective value. This checks feasibility as well as optimality. We have added these dataset and evaluation details to Appendix D.5 of the revised version of our paper.

Reference: [1] Michailidis, K., Tsouros, D. and Guns, T., 2025. DCP-bench-open: Evaluating LLMs for constraint modelling of discrete combinatorial problems. arXiv preprint arXiv:2506.06052.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer aHKL,

Thank you again for your thoughtful and constructive feedback. Following your suggestions, we added comparisons of selection rules on identical candidate pools, comparisons at similar API costs, and an audit of source-reference reliability. We also directly measured the token usage and cost of refinement within COOPA and clarified the DCP-Bench dataset and evaluation protocol.

We would appreciate your feedback on whether these additional results and clarifications address your concerns. If any questions remain, we would welcome the opportunity to discuss them further during the remaining discussion period.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response to All Reviewers [1/3]

We thank the reviewers for their careful reading and constructive suggestions. We have incorporated the additional experimental results and accompanying descriptions into the revised paper, which are highlighted in blue.

COOPA aims to make LLM-based optimization modeling more accurate, easier to inspect, and adaptable to different solvers. It combines explicit mathematical formulations, source references, iterative refinement, and specialized optimizer agents in a single framework. Parameters, decision variables, the objective, and constraints are explicitly represented and linked to cited passages from the problem description. The framework evaluates and refines these formulations before passing the selected candidate to an appropriate optimizer agent.

Following the reviewers’ suggestions, we conducted three additional sets of experiments: [CQ1] comparing selection rules on the same candidate pools to assess the role of final selection; [CQ2] comparing COOPA with all baselines given more attempts at approximately matched API costs; and [CQ3] using an LLM judge to audit the source references associated with the generated formulations. We present these experiments below and address reviewer-specific questions in the individual responses.

[CQ1] Comparison between max-min, max-mean, last candidate, random selection for candidate selection.

The reviewers ask whether confidence-based selection adds value beyond self-refinement. We use Max-min as a simple default rule, motivated by the fact that a single incorrect variable domain, objective, or constraint can invalidate an otherwise sound formulation. Under this rule, high scores on other components do not offset a low score on one component. The comparison below examines how much the choice of selection rule affects success rates.

We compare four rules on exactly the same candidate pools.

  • Last Candidate serves as the self-refinement baseline: it returns the final formulation produced by COOPA’s iterative feedback and revision.
  • Max-min and Max-mean select the candidate with the highest minimum or mean component score, respectively.
  • Random Selection chooses without using the scores.

We also report First Candidate as the reference before refinement. The experimental setup and results are reported in Appendix D.7 of the revised paper.

Results:

Max-min and max-mean achieve similar macro-average success rates: 70.22% and 70.55%, respectively, on GPT-5.2, and 69.65% for both on Gemini-3-Flash. Both confidence-based rules achieve higher macro-averages than First Candidate, Last Candidate, and Random Selection on both backbones. This suggests that confidence scores are useful for candidate selection, while the choice between minimum and mean aggregation has relatively little effect in this evaluation.

The effect of returning the last refined candidate varies across backbones: it improves the macro-average from 67.90% to 68.87% on Gemini-3-Flash, while yielding a slightly lower score on GPT-5.2 (69.02% versus 69.74%).

Success rates (%). Bold indicates the highest reported mean in each column for each backbone, including ties.

ModelSelection ruleComplexLPIndustryORBWORMacro-average
GPT-5.2First Candidate53.40 ± 0.7275.00 ± 1.7380.83 ± 2.8969.74 ± 1.23
GPT-5.2Random Selection53.40 ± 1.1975.00 ± 2.6579.58 ± 1.9169.33 ± 0.69
GPT-5.2Last Candidate53.24 ± 1.1974.67 ± 3.0679.17 ± 1.9169.02 ± 1.28
GPT-5.2Max-mean54.66 ± 1.1977.00 ± 1.0080.00 ± 2.1770.55 ± 0.77
GPT-5.2Max-min (COOPA)54.82 ± 0.9976.67 ± 1.1579.17 ± 2.6070.22 ± 1.15
Gemini-3-FlashFirst Candidate50.87 ± 1.1977.00 ± 2.6575.83 ± 0.7267.90 ± 0.51
Gemini-3-FlashRandom Selection50.08 ± 0.5576.67 ± 2.0877.50 ± 1.2568.08 ± 0.96
Gemini-3-FlashLast Candidate51.18 ± 0.0076.67 ± 2.0878.75 ± 1.2568.87 ± 0.89
Gemini-3-FlashMax-mean52.45 ± 0.2777.33 ± 2.5279.17 ± 1.4469.65 ± 1.19
Gemini-3-FlashMax-min (COOPA)52.45 ± 0.2777.33 ± 2.5279.17 ± 1.4469.65 ± 1.19
AuthorsOpenReview ↗

Common Response to All Reviewers [2/3]

[CQ2] Does COOPA still outperform the baselines when they are given a comparable inference budget?

The reviewers ask whether giving the baselines more attempts would let them match COOPA's success rate at approximately the same total API cost. We therefore run each baseline independently K times and combine its final reported objective values by majority voting. We vary K separately for each baseline to examine how success rates change with API cost, including configurations with costs close to COOPA's.

For voting, each numeric prediction viv_i receives a support count equal to the number of numeric predictions vjv_j, including itself, satisfying ∣vi−vj∣<0.1|v_i-v_j|<0.1. We select the prediction with the largest count and break ties by choosing the latest run.

All methods in this comparison use GPT-6.1-Sol. Each baseline is evaluated on the objective value selected by voting, while COOPA is evaluated on its final selected output from one complete pipeline run, including internal refinement. Following our main evaluation, we report the macro-average success rate, giving equal weight to ComplexLP, IndustryOR, and BWOR, alongside the average API cost per problem in USD. The comparison of macro-average success rates and API costs across attempt budgets is reported in Appendix D.8 of the revised paper.

Results with GPT-6.1-Sol

COOPA achieves a macro-average success rate of 73.55% at an average API cost of $0.148 per problem.

Each cell reports macro-average success rate (%) / average API cost per problem (USD). Bold marks the selected approximately cost-matched baseline configurations. “—” indicates a K value not reported here.

BaselineK=1K=2K=3K=4K=5K=6
OptiMUS65.80 / 0.168—————
Chain-of-Experts67.95 / 0.06368.29 / 0.12468.29 / 0.186———
OR-LLM-Agent63.61 / 0.03665.26 / 0.07265.50 / 0.10965.66 / 0.145——
OptiTree (Fixed)68.80 / 0.02967.85 / 0.05168.90 / 0.07368.01 / 0.10168.33 / 0.12468.72 / 0.145

Increasing K brings limited gains: OR-LLM-Agent improves by 2.05 percentage points from K=1 to K=4, Chain-of-Experts gains 0.34 points and plateaus after K=2, and OptiTree shows no consistent improvement. COOPA remains ahead at comparable API costs, achieving a macro-average success rate of 73.55% at $0.148 per problem, compared with 68.72% for OptiTree ($0.145), 68.29% for Chain-of-Experts ($0.124), and 65.66% for OR-LLM-Agent ($0.145).

These results also show that COOPA retains its advantage over the evaluated baselines with GPT-6.1-Sol, extending our findings to a newer LLM backbone.

AuthorsOpenReview ↗

Common Response to All Reviewers [3/3]

[CQ3] How reliable are COOPA's source references?

The reviewers ask for quantitative evidence on the reliability of COOPA's source references. Our main evaluation uses the benchmarks' reference optimal objective values to measure problem-solving performance. To complement this evaluation, we audit whether COOPA's source references faithfully connect individual modeling elements to the original problem description.

We audit 19,898 source-reference entries from COOPA's generated formulations in one GPT-5.2 run across ComplexLP, IndustryOR, and BWOR. These references are attached to parameters, variables, objectives, and constraints, following the structure described in Section 3.2. We check whether each source quote is a substring of the original problem description. We also use GPT-6-Luna to assess the support each reference provides for its associated modeling element.

The judge receives the full original problem, the associated element's definition, and the reference's quote and note. It checks the quote and note against the original problem using a rubric with three labels:

  • Supported: the cited information supports the element's stated source attribution, directly or through a simple derivation from explicit facts. Faithful paraphrases are allowed.
  • Uncertain: the evidence is ambiguous or insufficient to judge the claimed connection between the reference and the element.
  • Unsupported: the reference is missing, irrelevant, misattributed, contradicts the original text, or attributes invented facts to it.

The rubric requires an evidence excerpt from the original problem for each supported judgment. This assessment focuses on the evidence cited for each modeling choice. The audit setup, results, and complete evaluation prompt are provided in Appendix D.9 of the revised paper.

Results

Reference entriesSubstring-match rateGPT-6-Luna: supportedGPT-6-Luna: uncertainGPT-6-Luna: unsupported
19,89888.3%78.8%18.0%3.2%

Percentages are rounded. Substring matching and the judge's labels are separate measurements.

Most references have a clear basis in the original problem: 88.3% of the source quotes appear as substrings of the problem description, and GPT-6-Luna judges 78.8% of the references to support the associated source attribution. These results quantify the reliability of the source links COOPA provides for inspecting its modeling choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.