- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
COOPA is a modular optimization-modeling agent with three components: schema-validated formulations linked to quoted source text; generation and evaluation of multiple history-informed candidate formulations; and dispatch to specialized solver agents. Its default selection rule chooses the candidate with the highest minimum self-assessed confidence across parameters, variables, objectives, and constraints. Evaluation on three benchmarks and eight backbones reports 64.8% average accuracy, versus 61.6% for the strongest baseline by that aggregate measure, with COOPA leading on six backbones. Iteration, repeated-run, and routing studies supplement the main comparison. Accuracy is measured through agreement with reference objective values.
Critical strengths
- The architecture explicitly addresses executable but semantically incorrect formulations. Formulation assessment occurs before solver-code generation, rather than relying exclusively on execution failures to trigger correction. Structured modeling elements and source quotations also make individual modeling choices inspectable. The running example illustrates the intended benefit: an executable sum-of-efficiencies objective is revised into the appropriate bottleneck-throughput objective. This is a relevant failure mode that ordinary debugging alone would not necessarily catch.
- There is meaningful empirical support beyond a single backbone or demonstration. The paper compares eight backbones, repeats selected comparisons three times, and studies refinement-budget sensitivity. It also reports a routing comparison on 164 DCP-Bench instances, where the full system improves over always using the mathematical optimizer by 8.74 percentage points, with a positive problem-clustered confidence interval. These checks strengthen the evidence that parts of the architecture are useful, even though the benefits are not universal.
Critical weaknesses
- The experiments do not isolate the value of max-min confidence selection from additional refinement and inference expenditure. The principal ablation compares the first formulation against selection from three history-refined formulations. That combines better candidate generation with candidate selection; it does not establish that max-min outperforms simply taking the last candidate, random selection from the same pool, or another selector. The confidence–accuracy correlation does not resolve this confounding. This matters because the cost study finds approximately seven times the API cost of OR-LLM-Agent on the sampled Gemini-2.5-Flash workloads, while the overall accuracy advantage is modest.
- The claimed human-verification benefit remains an architectural proposition rather than a validated outcome. The paper does not measure whether provenance and confidence explanations improve human error detection or reduce verification time, and it does not separately audit the accuracy and completeness of source references. Its confidence analysis establishes some association with improvement, but not calibrated probabilities of correctness. Consequently, the outputs may be useful inspection aids, but the evidence does not yet establish that they make human reliance on the generated models more reliable.
Questions for authors
- Does COOPA genuinely detect semantic formulation errors, or does it mainly benefit from additional sampling and refinement? The current ablation compares the full procedure against using only the first candidate, which conflates additional candidate generation, self-refinement, and confidence-based selection. Could the authors compare against selecting the last candidate, random selection, or best-of- under alternative selectors using exactly the same candidate pool and inference budget?
- How reliable is the confidence evaluator specifically at identifying semantic errors? Could the authors evaluate whether low confidence correctly localizes the erroneous component (parameter, variable, objective, or constraint) using annotated formulation errors independently? The current end-to-end accuracy results do not establish whether the evaluator itself can reliably detect semantic mistakes.
- What happens when the model is confidently wrong? Since both formulation generation and correctness assessment are performed by LLMs without an external semantic verifier, can the authors quantify cases where an incorrect formulation receives uniformly high confidence and is therefore not repaired?
4 comments
Follow-up on Our Rebuttal
Dear Reviewer ofnf,
Thank you again for your detailed and constructive comments. To address your concerns about refinement, selection, and inference cost, we added comparisons that hold the candidate pools fixed while changing the final selection rule, together with comparisons at similar API costs. We also audited source-reference reliability and clarified how the references support component-by-component inspection, how confidence scores are used, and the scope of the evidence supporting these uses.
We would appreciate hearing whether these results and explanations address your concerns, and whether there are remaining points we could clarify during the discussion.
Thank you for the selection ablations, cost-matched comparisons, and source-reference audit. These strengthen the evidence for the overall workflow. However, max-min shows no clear advantage over max-mean, and the audit establishes source support rather than formulation correctness. I maintain my rating of 3: the additional evidence is encouraging, but the methodological contribution remains modest for a stronger recommendation.
Response to Reviewer ofnf
Dear Reviewer ofnf,
Thank you for carefully considering our additional experiments and for your constructive feedback throughout the discussion. We respect your assessment and your decision to maintain your rating. We would like to briefly clarify the two points you raised, in case our earlier explanations were not sufficiently clear.
First, max-min is a reasonable default, motivated by the fact that a single incorrect modeling component can invalidate an entire formulation. We do not claim that it should outperform every alternative. Both max-min and max-mean achieve higher average success rates than Last Candidate and Random Selection, supporting the effectiveness of our confidence-guided selection approach across different aggregation rules.
Second, as explained in [Q3], the benchmarks provide only reference objective values, not ground-truth formulations. An LLM tasked with auditing formulation correctness must therefore independently resolve the same semantic modeling questions posed by the natural-language problem description. Our GPT-6.1-Sol experiments demonstrate that even frontier models continue to make errors on these tasks, meaning that another LLM's judgment cannot automatically be treated as reliable ground truth. This is fundamentally different from source-reference auditing, where an LLM checks whether the cited text supports the attribution of an individual modeling element, such as a variable or objective.
More importantly, agreement with the reference optimum is already a strong indicator of formulation correctness under the existing benchmark protocols. Most benchmark problems are specified precisely enough to admit a single intended mathematical interpretation. In these cases, reproducing the reference optimum provides strong evidence that the generated formulation captures the intended problem, even without directly comparing formulations. For the relatively few problems whose descriptions admit multiple valid interpretations and, consequently, multiple defensible formulations, there may be no uniquely defined ground-truth formulation against which to assess correctness. In such cases, requiring agreement with a particular formulation would risk rejecting valid alternatives, whereas objective-value comparison provides a consistent and operationally well-defined evaluation criterion under the established benchmark protocols. Establishing reliable formulation-level correctness labels beyond these existing evaluations would therefore require substantial additional annotation, explicit treatment of ambiguity, and independent validation.
Response to Reviewer ofnf [1/1]
Thank you for your careful review and constructive suggestions. We have added comparisons of selection rules on the same candidate pools [CQ1] and performance at comparable API costs [CQ2], together with a source-reference audit [CQ3], to address your concerns. We also respond below to your questions about verification and confidence reliability.
[Q1] Do the gains come from refinement, final selection, or additional computation?
Following your suggestion, the new comparison in [CQ1] holds the generated candidate pools and refinement process fixed while changing only the final selection rule. Max-min therefore receives exactly the same candidates as Last Candidate, Random Selection, and max-mean. It achieves higher macro-average success rates than Last Candidate and Random Selection on both backbones, including gains of 1.20 percentage points over Last Candidate on GPT-5.2 and 0.78 points on Gemini-3-Flash. Max-mean achieves similar results. These direct comparisons support the value of confidence-guided final selection on an unchanged refinement history, with the benefit observed under both minimum and mean aggregation.
To test whether additional baseline attempts close the performance gap at a similar cost, [CQ2] compares all four baselines using GPT-6.1-Sol and reports their actual API costs. For example, OR-LLM-Agent receives four independent attempts combined by majority voting, achieving 65.66% macro-average success at $0.145 per problem. COOPA achieves 73.55% at $0.148 with one complete pipeline run, including its internal refinement. COOPA also retains the highest macro-average success rate across the four baseline comparisons at comparable costs. This provides evidence that its advantage persists when competing methods receive a similar API budget for additional attempts.
[Q2] Do source references and confidence explanations improve human verification, and how are the references themselves evaluated?
Source references connect individual modeling elements to the problem description, allowing a formulation to be inspected component by component. For example, an interface could display each parameter, variable, objective, and constraint alongside its cited passage and the corresponding confidence explanation. For example, a reviewer could check whether a capacity constraint uses the limit stated in the cited passage, with the evaluator’s explanation available to help examine the modeling choice. Evaluating the effects of these features on human error detection and verification time requires dedicated user studies, which are beyond the scope of this paper, as discussed in Appendix F.2.
The new audit in [CQ3] provides quantitative evidence on the reliability of these source links, with most references judged to support their associated source attributions. The audit evaluates the references supplied with the generated elements; whether they cover all source information needed to verify the full formulation would require a separate completeness assessment.
COOPA uses the evaluator's scores to guide refinement and rank candidate formulations for the same problem. These scores are not calibrated probabilities of correctness; for example, a score of 90 does not imply a 90% probability of correctness. The same-pool comparison in [CQ1] directly evaluates their use for final selection and supports their value in choosing among candidate formulations.
[Q3] Do low confidence scores identify incorrect components, and how often does an incorrect formulation receive uniformly high confidence and remain uncorrected?
The three main benchmark datasets used in our evaluation provide reference optimal objective values but no reference mathematical formulations. We therefore follow the benchmark protocol and check whether the generated model produces the reference optimum. This is a useful indicator of successful modeling: errors in the variables, objective, or constraints can change the optimal value, so matching the reference value tests whether the model reproduces the expected optimization result.
The new comparison in [CQ1] further tests whether confidence helps select formulations that achieve this result. With candidate pools held fixed, both max-min and max-mean achieve higher macro-average success rates than Last Candidate and Random Selection on both backbones. These gains provide evidence that the evaluator's assessments contain useful information for selecting successful candidates.
Quantifying how accurately low confidence identifies erroneous components, and how often semantic errors receive uniformly high scores and persist through refinement, would require an additional audit with independently annotated formulations across refinement rounds. Such an audit would build on the selection evidence in [CQ1] to assess how consistently the evaluator identifies specific modeling errors and how often subsequent revisions correct them.