acceptodds
Under review as a conference paper at ICLR 2027

TreasuryOpt: Benchmarking LLMs on Generating Reliable Optimization Programs

Abstract

Large language models (LLMs) are increasingly explored for mathematical optimization, yet existing evaluations rarely test whether they can construct reliable optimization programs that remain robust across unseen instances. We introduce TreasuryOpt, a benchmark evaluating LLMs on generating reliable optimization programs grounded in enterprise treasury operations, comprising four scenarios, 28 modeling cases, and 596 hidden test instances. For each modeling case, a system must construct a fixed program from its textual requirements, which is then evaluated across unseen instances without further model intervention. Our solver-verified evaluation protocol then extracts the returned decision variables and independently recalculates constraint satisfaction and objective optimality against solver-certified reference bounds. Extensive experiments across six frontier LLMs—under standalone and agentic settings across solver-hidden and solver-aware conditions—reveal a persistent feasibility–optimality gap: systems face considerable difficulty satisfying domain constraints, with success rates collapsing further when requiring solver-certified optimality. Furthermore, while interactive coding harnesses enhance solution quality via iterative self-debugging, they incur orders of magnitude higher generation costs with only limited performance gains. These results demonstrate that reliable optimization with LLMs requires progressing beyond surface-level executability toward mathematically sound and generalizable program generation.

Primary area:
optimization
Keywords:
Large Language Models, Mathematical Optimization, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JeYROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TreasuryOpt evaluates the generation of reusable optimization programs, rather than solutions for individual instances. A system receives a textual modeling specification and submits a fixed program, which is then tested on hidden inputs without further LLM intervention. The benchmark contains four treasury scenarios, 28 modeling cases, and 596 hidden instances. Independent checkers recompute feasibility and objective quality from returned decisions against solver-certified references. A case passes only when every associated hidden instance passes. Across six models, standalone and agentic settings, and solver-information conditions, the paper reports substantial differences between constraint correctness and objective validation, together with expensive and uneven benefits from agentic generation.

Critical strengths
  • The frozen-program, hidden-instance design evaluates a practically important capability. Requiring one program to implement a specification across changing inputs tests whether the generated code captures reusable decision logic. Prohibiting further LLM intervention during hidden testing also prevents instance-specific repair from concealing failures of the submitted program. This makes the evaluation unit well aligned with the intended use of optimization software as a repeatedly deployed decision tool.
Critical weaknesses
  • The benchmark’s construction and ground-truth assurance are difficult to evaluate by myself from the manuscript. The construction section describes expert review, production-derived rules, synthetic instances, and Gurobi references, but does not provide a complete worked modeling case, explicit instance-generation ranges, numerical checker tolerances, or reference optimality-gap statistics. Therefore, I have limited evidence for assessing whether specifications unambiguously determine the intended semantics and how computational or numerical difficulty is distributed. For a benchmark whose central contribution is reliable verification, these details are consequential rather than merely implementation conveniences.
Questions for authors
  • What exactly constitutes the ground-truth specification for each modeling case? The paper states that each case contains a textual specification and is derived from production business rules, but it is unclear whether the full mathematical semantics, e.g., variables, constraints, objective hierarchy, and edge cases, are explicitly specified to the evaluated model. Could the authors provide one complete example showing the textual specification, reference formulation, and corresponding checker?
  • How is ambiguity in the textual specifications handled? Since generated programs are judged against a fixed reference implementation, how do the authors ensure that a program implementing a reasonable alternative interpretation of an underspecified business requirement is not incorrectly counted as a failure? Was semantic unambiguity independently verified for all 28 cases?
  • Can the benchmark distinguish an incorrect formulation from insufficient optimization? A program can fail Validation either because it encodes the wrong optimization problem or because it encodes the correct problem but fails to solve it sufficiently well within the computational budget. Can the authors separate these two failure modes? This distinction seems important if TreasuryOpt is intended primarily as a benchmark of optimization-program generation.
Reviewer_CqRMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TreasuryOpt, a benchmark for evaluating whether LLMs can generate reusable optimization programs from enterprise treasury specifications. It contains four scenarios, 28 modeling tasks, and 596 hidden instances. A fixed program is generated for each task, with no further LLM-based repair during hidden testing. The evaluator independently checks the interface, business constraints, and objective quality using the returned decisions. Correctness (C) requires the program to pass execution and constraint checks on every hidden instance of a task. Validation (V) additionally requires it to match the reference objective within the prescribed tolerance. The authors compare six models under direct generation, agent-assisted generation, and different solver-information settings. They also report the fraction of tasks passed by at least one of three candidate programs. Under the benchmark's tasks and budgets, the results show a substantial gap between satisfying constraints and meeting the reference objective. Agents generally improve performance but increase generation costs. The appendix examines constraint violations, objective gaps, and debugging trajectories. These analyses illustrate misunderstood business rules, inefficient solving, and limitations of local self-testing.

Critical strengths

S1. The benchmark addresses practical reliability needs in enterprise treasury operations and has potential application value.

The paper addresses a meaningful problem and provides useful treasury examples. Its tests cover fund conservation, account capacity, lexicographic objectives, and settlement dependencies. It independently checks the returned decisions. This design can expose errors in implementing business rules that a successful run on one instance may miss. Program-level evaluation has precedents, but the strict business and numerical requirements of treasury operations could still provide a useful addition to existing benchmarks. (pp. 3–5, Sections 3.1–3.3)

S2. The paper provides useful diagnoses of business-rule failures, beyond aggregate model scores.

The paper reports C/V, first-attempt performance, and best-of-three candidate coverage. The appendix further breaks down constraint violations and objective gaps. Examples include cash-pool capacity and balance errors, overdrafts during settlements within the same layer of the dependency graph, and an agent changing test assertions to accommodate incorrect fund flows. These examples make the practical consequences of failures clear. Cases where boundary tests and numerical checks successfully repair programs also show the benefits of agents. Together, these analyses help identify what needs careful verification when developing optimization programs. (p. 6, Table 2; pp. 15–18, Tables 9–11 and Sections A.2.1 and A.2.3)

Critical weaknesses

W1. The paper addresses a meaningful problem and provides useful treasury examples. However, its core evaluation ideas have precedents, and it does not clearly explain what new research value the treasury setting adds.

The paper emphasizes evaluating the reliability of a fixed program across new inputs, rather than checking whether it solves a single instance. Prior work already evaluates this capability. CO-Bench (Sections 3.1–3.3) and ALE-Bench (Sections 3.2 and A.3.3) both evaluate final programs on unseen test instances. CO-Bench also requires valid results on all test instances of a problem for its Valid Solution metric. FrontierOR (Sections 3.2 and 4) independently checks constraints and objective quality. OptiHive (Methodology and Experimental Setup) also studies the unreliability of generated programs and their self-tests.

The potential added value lies in treasury-specific requirements, such as fund conservation, objective priorities, and settlement order. A specialized benchmark can be valuable. However, the paper does not sufficiently explain which difficulties these requirements introduce that existing benchmarks do not cover. Which errors can only be exposed by these tasks? The paper therefore needs to distinguish findings within this domain from broader conclusions. (p. 4, Section 3.2) The abstract's characterization of the gains from agents also needs revision.

W2. The experiments demonstrate failures, but several main explanations lack sufficient supporting evidence.

2.1 A higher pass rate for constraint satisfaction than for meeting the reference objective is not surprising. The latter metric includes all requirements of the former and adds an objective requirement. The size of the gap and the business rules that are difficult to handle are still worth studying. The appendix also provides useful examples. However, prior work has already discussed that executable programs can return incorrect solutions and that self-tests can miss errors. The paper needs to explain what these experiments add to our understanding of the causes of failure. A few successful or unsuccessful debugging cases do not establish that a particular checking strategy is generally more effective. (p. 5, Section 3.3; appendix analyses on pp. 15–18)

2.2 The authors argue that providing solver information prompts agents to reformulate their optimization models and may introduce new modeling errors. However, score differences with and without solver information only show that the outcomes differ. They do not explain why. The current evidence does not sufficiently connect solver hints, reformulation, and specific errors. This explanation needs concrete task-level analysis. Otherwise, it should be presented as a possible cause that remains to be verified. (pp. 7–8, Sections 4.2 and 4.4)

2.3 The abstract states that agents incur high generation costs but offer limited gains. Yet Table 2 shows that, under Solver Hidden, Qwen's V@1 rises from 7.14% with direct generation to 46.43% with OpenClaw, while Kimi's rises from 0% to 35.71%. These gains are substantial. The experiments support increased costs, but this alone does not justify a general claim of limited gains. The paper should distinguish settings with small gains from those with substantial gains. Since the generated programs can be reused, a high one-time generation cost is also insufficient to determine whether their long-term use is worthwhile. (p. 1, Abstract; p. 6, Table 2; pp. 7–8, Section 4.3)

Questions for authors

Q1. What does this benchmark reveal beyond prior work? Can the authors use concrete tasks to explain which treasury requirements are not adequately covered by existing benchmarks? Do the experiments reveal distinct errors arising from these requirements?

Q2. What broader conclusions do these tasks support? Beyond showing that the evaluated models still fail on treasury tasks, what new conclusions do the experiments support? Which findings apply to other optimization tasks, and what evidence supports that extension?

Q3. What direct evidence supports the claim that solver hints cause new modeling errors? Can the authors use specific tasks to show how the hints affect formulation choices and how the errors arise? If the evidence consists only of score changes, should this explanation be framed as one possible cause?

Q4. Which results support the claim that agents offer limited gains? Qwen and Kimi improve substantially in some settings. How do the authors reconcile these results with the abstract? Does the cost conclusion concern only one-time program generation, rather than the benefits of reusing the program? Please clarify this distinction.

Reviewer_yxsiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TreasuryOpt, a benchmark for evaluating whether LLMs can generate reliable optimization models that remain robust across unseen data instances. They propose this benchmark because conventional evaluations are usually confined to instance-level assessment (namely testing if a model can get the optimal solution under fixed parameters). This benchmark in grounded in enterprise treasury operations, resulting in 596 test instances. Their experiments on 6 models show that current LLMs fall short in generating reliable optimization models.

Critical strengths
  1. This paper is well-organised and easy to follow;

  2. This paper is well-motivated, and the topic is interesting but under-explored in LLMs-based optimization modeling;

Critical weaknesses
  1. The contribution is very limited (in terms of size and scope): this benchmark is very small in size, with merely 28 test cases (although leading to 596 instances in total) and all of them are from a single domain (namely company treasury operations). The value of such a small benchmark is limited, and is not a good fit to be published in top venues such as ICLR (maybe it should be submitted to ICAIF).

  2. The evaluated baselines are limited: although the authors claim that they conduct "extensive experiments", they only evaluated six bare LLMs and their openClaw variants. However, classical and popular methods such as Multi-Agent based methods including Chain-of-Experts and OptiMUS, fine-tuned methods such as ORLM, StepORLM, OptMATH, LLMOPT, SIRL, SAGE were not evaluated.

  3. The added scientific value over existing benchmarks is not demonstrated: The main change is to evaluate a fixed program across multiple hidden inputs and require every instance to pass. However, no controlled comparison shows what this reveals beyond existing optimization benchmarks or instance-level evaluation. In particular, the authors do not establish whether existing benchmarks would also reveal the feasibility–optimality gap, sampling benefits, and agentic costs. Stricter scoring alone does not demonstrate a new capability being measured, leaving the benchmark’s scientific contribution unclear.

  4. The experiments show that models fail, but do not explain what makes the problems difficult: For example, do models struggle because they must prioritize several objectives, or because transactions depend on one another? The paper does not test simpler versions of the same problems, such as using one objective or removing transaction dependencies, while keeping other factors fixed. Counting timeouts, constraint violations, and suboptimal solutions describes the failures without answering these questions. Consequently, the findings provide limited insight into which specific capabilities current models lack.

Questions for authors

Please check the weak points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.