- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper evaluates stochastic optimization workflows under two information conditions: NL2Code, where the relevant stochastic parameters are supplied, and Workflow, where they must be estimated from 100 observations. The benchmark contains 147 NL2Code tasks and 441 Workflow tasks across 21 human-verified executable protocols, including distribution changes and corrupted observations. It also introduces neuron-guided task selection. Across eight evaluated models, the strongest systems achieve over 99% tolerance accuracy with supplied parameters, but approximately 7% capped objective-value error in Workflow.
Critical strengths
- The matched information conditions and same-data reference make the benchmark diagnostically useful. The clean L1 tasks preserve the underlying optimization problem while changing whether stochastic parameters are supplied or inferred. Adding a reference that uses exactly the agent’s observations is particularly valuable: it prevents every discrepancy from the latent optimum from automatically being attributed to faulty optimization. This is a more informative design than comparing unrelated “easy” and “hard” problem collections.
- The executable protocols and construction controls are substantive benchmark assets. Each protocol includes a schema, simulator, and reference solver reviewed by two PhD-level reviewers. The construction study also compares neuron-based selection against embedding and random alternatives while controlling candidate batches and generation budgets, with diversity assessed on held-out models.
Critical weaknesses
- The principal metric does not establish decision quality, despite the paper’s decision-making framing. Appendix F.1 provides a particularly clear example: both the true optimizer and the empirical optimizer choose the same optimal order quantity, , yet the empirical solution receives a 7.37% error because its estimated profit differs from the true expected profit. Thus, a genuinely optimal operational decision can be counted as an inaccurate answer. The results support claims about optimal-value estimation, but not directly about poor decisions, infeasibility, or decision regret.
- The paper’s “observation-to-decision” framing is somewhat broader than the actual task. In the Workflow setting, the optimization structure (the decision problem, objective, and constraints) is already specified in natural language; the agent mainly needs to estimate hidden stochastic parameters from 100 observations and then solve the resulting optimization problem. Thus, the benchmark does not test whether an agent can infer the underlying decision problem itself from operational data.
Questions for authors
- Could the authors clarify the intended scope of “from data to decisions”? In particular, do you view ORWorkerBench primarily as evaluating stochastic-parameter estimation followed by optimization, rather than learning the optimization problem (e.g., objectives, constraints, or decision structure) from data?
- The benchmark evaluates the error in the reported optimal objective value rather than the quality of the resulting decision. As illustrated by the paper’s own example, the empirical and true problems can produce the same optimal decision while still receiving a sizable objective-value error. Could the authors report decision-level metrics, such as decision regret under the true distribution or the fraction of cases in which the inferred and true optimal decisions agree?
- The strongest models perform very close to the same-data estimate-then-optimize reference. How should we interpret the remaining approximately 7% error as an LLM limitation rather than ordinary finite-sample estimation error? Can the authors provide stronger statistical estimation baselines or an oracle/statistical benchmark that better quantifies the achievable performance given only 100 observations?
- All Workflow tasks use exactly 100 observations. How does the observation-to-decision gap change with sample size (e.g., 25, 50, 200, or 500 observations)? Without such a comparison, it is difficult to determine whether the reported gap primarily reflects limitations of LLM reasoning or simply the particular finite-data regime chosen by the benchmark.
5 comments
Response to R. GMxd (Page 2/2)
W2 & Q1: The paper’s “observation-to-decision” framing is somewhat broader than the actual task. In the Workflow setting, the optimization structure (the decision problem, objective, and constraints) is already specified in natural language; the agent mainly needs to estimate hidden stochastic parameters from 100 observations and then solve the resulting optimization problem. Thus, the benchmark does not test whether an agent can infer the underlying decision problem itself from operational data.
Thank you for this great point. The main scope of our ORWorkerBench is to evaluate observation-driven stochastic optimization within a specified decision problem. The operational objective and constraints are supplied in natural language; agents must infer the hidden stochastic quantities and use them to formulate and solve the optimization problem. We do not evaluate discovery of the objective or constraint structure from raw data. Under this setting, agents identify and estimate decision-relevant stochastic parameters from finite observations before formulating and solving the optimization model.
Regarding the concern about inferring the underlying decision problem from operational data, we would like to share our thoughts about the design choice. While the decision problem, objective, and constraints are provided in natural language, translating these descriptions into appropriate OR formulations and executable solver code remains a challenging task that has received substantial attention from the research community. Building upon the traditional NL2Code setting, we <u>extends OR problem solving to an observation-driven operational setting, where decision-relevant stochastic parameters are not explicitly given but must instead be inferred from historical observations</u>. Therefore, we believe the work represents a meaningful and challenging step beyond the traditional NL2Code paradigm.
We totally agree with your insights on the research direction. We will clarify the scope of "from data to decisions" in the Introduction section of the camera-ready version and discuss problem discovery from real-world document or messy operational data as a valuable future extension to explore.
Q3: The strongest models perform very close to the same-data estimate-then-optimize reference. How should we interpret the remaining approximately 7% error as an LLM limitation rather than ordinary finite-sample estimation error? Can the authors provide stronger statistical estimation baselines or an oracle/statistical benchmark that better quantifies the achievable performance given only 100 observations?
Q4: All Workflow tasks use exactly 100 observations. How does the observation-to-decision gap change with sample size (e.g., 25, 50, 200, or 500 observations)? Without such a comparison, it is difficult to determine whether the reported gap primarily reflects limitations of LLM reasoning or simply the particular finite-data regime chosen by the benchmark.
Thank you for this point. We view the same-data reference as a comparator for the effect of finite-sample variability on objective-value estimation**.** Thus, the strongest models’ proximity to this reference suggests that their remaining approximately 7% error should not be attributed entirely to LLM limitations. Our additional experiments, as shown in Table B, increasing the number of observations from 100 to 400 consistently reduces error across five models. This sweep covers Regular tasks only and reuses historical runs with 100 observations. Interestingly, while the best-performing model, Astra, outperforms the same-data reference baseline with 100 observations, it falls behind the reference at both 200 and 400 observations, despite the consistent error reduction brought by larger observation budgets.
Table B. Workflow objective-value error with 100, 200, and 400 observations, on Regular Split.
| Model | 100 obs. | 200 obs. | 400 obs. |
|---|---|---|---|
| GPT-4o | 19.35 | 18.13 | 18.11 |
| GPT-5.2 | 6.58 | 4.50 | 3.26 |
| GPT-5.6 Luna | 6.33 | 4.59 | 3.69 |
| GPT-5.6 Terra | 6.15 | 3.80 | 3.07 |
| GPT-5.6 Sol | 5.76 | 3.64 | 2.77 |
| GPT-6 Astra | 5.76 | 3.65 | 2.83 |
| Same-data reference | 5.82 | 3.55 | 2.79 |
We will clarify that this reference contextualizes finite-data effects; it does not establish the best achievable statistical performance or precisely decompose all error sources. Stronger statistical estimators remain a valuable direction for further evaluation.
Reference:
[1] Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, and Yinyu Ye. Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling. Advances in Neural Information Processing Systems (NeurIPS 2025), Volume 38, 2025.
Thank you for the additional experiments and clarifications. The sample-size sweep and the more careful interpretation of finite-sample error meaningfully address my concerns.
My main remaining concern is decision quality. The decision-consistency analysis is useful, but disagreement with the paired NL2Code decision does not necessarily imply higher regret, particularly when multiple optimal solutions exist. Non-uniqueness does not prevent evaluating a returned decision under the true distribution. Could you report feasibility and decision regret on at least a subset of tasks, ideally alongside the same-data reference?
Response to R. GMxd (Page 1/2)
Thank you for your insightful comments! Below are our detailed responses.
Thank you for highlighting this important distinction. We agree that objective-value error does not equal to decision quality, as acknowledged in our Limitations section. In some given cases, such as the example in Appendix F.1, as you point out, the empirical optimizer selects the same optimal decision but still incurs 7.37% objective-value error due to finite-sample estimation.
The main reason we adopt objective-value-based evaluation is to provide a consistent, automatically verifiable, and scalable metric across different OR problems, as some prior works also did [1]. Decisions can take different forms across OR problems, ranging from simple quantities to multi-stage decisions and complex policies. Moreover, optimal decisions may not be unique, making it impractical to enumerate all optimal solutions for direct decision matching. Finding the ground-truth decisions requires substantial human involvement in problem-specific decision extraction, feasibility checking, and verification, thus limiting the scalability of benchmark construction. Therefore, we use objective-value error as our primary metric.
We recognize your concerns about this limitation, so we conducted an additional decision-level analysis on the paired NL2Code and Workflow tasks under the Python-only setting with 100 observations. Specifically, we used an LLM judge (GPT-6.1 Sol) to extract the decisions from the original agents' execution trajectories and grouped the paired tasks based on whether their decisions were consistent. We then compared the original Workflow objective-value errors between these two groups. Given that the strongest models achieve near-perfect performance on NL2Code tasks, we expect Workflow decisions that differ from their NL2Code counterparts to be associated with higher objective-value errors.
As shown in Table A, objective-value error is consistently around 2–3 times higher when the Workflow decision differs from its NL2Code counterpart, where the strongest models achieve near-perfect performance. While decision consistency is not equivalent to decision regret, these results provide additional evidence that the observed observation-to-decision gap is associated not only with objective-value estimation errors, but also with differences in the decisions made by the models. This suggests that our current metric can provide reference to the decision quality difference.
Table A. Original Workflow objective-value errors on paired L1 tasks, grouped by whether the LLM-judge-extracted decisions agree with those of their NL2Code counterparts.
We will include the complete additional decision-level results in the revised manuscript and further clarify the distinction between objective-value error and decision quality. We really appreciate you pointing this out.