- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TreasuryOpt evaluates the generation of reusable optimization programs, rather than solutions for individual instances. A system receives a textual modeling specification and submits a fixed program, which is then tested on hidden inputs without further LLM intervention. The benchmark contains four treasury scenarios, 28 modeling cases, and 596 hidden instances. Independent checkers recompute feasibility and objective quality from returned decisions against solver-certified references. A case passes only when every associated hidden instance passes. Across six models, standalone and agentic settings, and solver-information conditions, the paper reports substantial differences between constraint correctness and objective validation, together with expensive and uneven benefits from agentic generation.
Critical strengths
- The frozen-program, hidden-instance design evaluates a practically important capability. Requiring one program to implement a specification across changing inputs tests whether the generated code captures reusable decision logic. Prohibiting further LLM intervention during hidden testing also prevents instance-specific repair from concealing failures of the submitted program. This makes the evaluation unit well aligned with the intended use of optimization software as a repeatedly deployed decision tool.
Critical weaknesses
- The benchmark’s construction and ground-truth assurance are difficult to evaluate by myself from the manuscript. The construction section describes expert review, production-derived rules, synthetic instances, and Gurobi references, but does not provide a complete worked modeling case, explicit instance-generation ranges, numerical checker tolerances, or reference optimality-gap statistics. Therefore, I have limited evidence for assessing whether specifications unambiguously determine the intended semantics and how computational or numerical difficulty is distributed. For a benchmark whose central contribution is reliable verification, these details are consequential rather than merely implementation conveniences.
Questions for authors
- What exactly constitutes the ground-truth specification for each modeling case? The paper states that each case contains a textual specification and is derived from production business rules, but it is unclear whether the full mathematical semantics, e.g., variables, constraints, objective hierarchy, and edge cases, are explicitly specified to the evaluated model. Could the authors provide one complete example showing the textual specification, reference formulation, and corresponding checker?
- How is ambiguity in the textual specifications handled? Since generated programs are judged against a fixed reference implementation, how do the authors ensure that a program implementing a reasonable alternative interpretation of an underspecified business requirement is not incorrectly counted as a failure? Was semantic unambiguity independently verified for all 28 cases?
- Can the benchmark distinguish an incorrect formulation from insufficient optimization? A program can fail Validation either because it encodes the wrong optimization problem or because it encodes the correct problem but fails to solve it sufficiently well within the computational budget. Can the authors separate these two failure modes? This distinction seems important if TreasuryOpt is intended primarily as a benchmark of optimization-program generation.