acceptodds
Under review as a conference paper at ICLR 2027

ORWorkerBench: Benchmarking LLMs from Data to Decisions under Uncertainty

Abstract

Operations Research increasingly provides the decision layer for AI systems that must allocate resources, schedule work, manage inventory, or act under uncertainty. Recent optimization benchmarks show strong performance from frontier LLMs and agents, but many begin from a specified optimization problem, where the stochastic quantities required by the model are already provided. In real operational settings, those quantities often have to be inferred from finite observations before optimization can begin. We introduce ORWorkerBench, a benchmark for evaluating this observation-to-decision setting. Each underlying decision problem is evaluated under matched conditions: in NL2Code, the relevant stochastic parameters are given explicitly, while in Workflow, the agent must infer them from 100 observations before constructing and executing the optimization procedure. Our benchmark contains 147 NL2Code tasks and 441 Workflow tasks across 21 human-verified executable protocols. We additionally use neuron-guided synthesis to reduce behavioral redundancy during benchmark construction; compared with embedding-based selection, it produces more diverse activation patterns across four held-out models. Our evaluation of six proprietary and two open-weight LLMs reveals a substantial observation-to-decision gap. The strongest models achieve over 99% tolerance accuracy when the stochastic quantities are supplied, yet incur roughly 7% mean capped error when those quantities must instead be recovered from observations. Moreover, even the strongest systems perform close to a simple same-data reference that estimates the required stochastic quantities from the same observations and passes them to a verified solver. These results show that strong performance on specified optimization problems does not yet translate into reliable decision making from finite observations.

Primary area:
datasets and benchmarks
Keywords:
LLMs evaluation, Operations research, Observation-to-decision, Neuron-guided data synthesis

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_GMxdOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper evaluates stochastic optimization workflows under two information conditions: NL2Code, where the relevant stochastic parameters are supplied, and Workflow, where they must be estimated from 100 observations. The benchmark contains 147 NL2Code tasks and 441 Workflow tasks across 21 human-verified executable protocols, including distribution changes and corrupted observations. It also introduces neuron-guided task selection. Across eight evaluated models, the strongest systems achieve over 99% tolerance accuracy with supplied parameters, but approximately 7% capped objective-value error in Workflow.

Critical strengths
  • The matched information conditions and same-data reference make the benchmark diagnostically useful. The clean L1 tasks preserve the underlying optimization problem while changing whether stochastic parameters are supplied or inferred. Adding a reference that uses exactly the agent’s observations is particularly valuable: it prevents every discrepancy from the latent optimum from automatically being attributed to faulty optimization. This is a more informative design than comparing unrelated “easy” and “hard” problem collections.
  • The executable protocols and construction controls are substantive benchmark assets. Each protocol includes a schema, simulator, and reference solver reviewed by two PhD-level reviewers. The construction study also compares neuron-based selection against embedding and random alternatives while controlling candidate batches and generation budgets, with diversity assessed on held-out models.
Critical weaknesses
  • The principal metric does not establish decision quality, despite the paper’s decision-making framing. Appendix F.1 provides a particularly clear example: both the true optimizer and the empirical optimizer choose the same optimal order quantity, Q=91Q=91, yet the empirical solution receives a 7.37% error because its estimated profit differs from the true expected profit. Thus, a genuinely optimal operational decision can be counted as an inaccurate answer. The results support claims about optimal-value estimation, but not directly about poor decisions, infeasibility, or decision regret.
  • The paper’s “observation-to-decision” framing is somewhat broader than the actual task. In the Workflow setting, the optimization structure (the decision problem, objective, and constraints) is already specified in natural language; the agent mainly needs to estimate hidden stochastic parameters from 100 observations and then solve the resulting optimization problem. Thus, the benchmark does not test whether an agent can infer the underlying decision problem itself from operational data.
Questions for authors
  • Could the authors clarify the intended scope of “from data to decisions”? In particular, do you view ORWorkerBench primarily as evaluating stochastic-parameter estimation followed by optimization, rather than learning the optimization problem (e.g., objectives, constraints, or decision structure) from data?
  • The benchmark evaluates the error in the reported optimal objective value rather than the quality of the resulting decision. As illustrated by the paper’s own example, the empirical and true problems can produce the same optimal decision while still receiving a sizable objective-value error. Could the authors report decision-level metrics, such as decision regret under the true distribution or the fraction of cases in which the inferred and true optimal decisions agree?
  • The strongest models perform very close to the same-data estimate-then-optimize reference. How should we interpret the remaining approximately 7% error as an LLM limitation rather than ordinary finite-sample estimation error? Can the authors provide stronger statistical estimation baselines or an oracle/statistical benchmark that better quantifies the achievable performance given only 100 observations?
  • All Workflow tasks use exactly 100 observations. How does the observation-to-decision gap change with sample size (e.g., 25, 50, 200, or 500 observations)? Without such a comparison, it is difficult to determine whether the reported gap primarily reflects limitations of LLM reasoning or simply the particular finite-data regime chosen by the benchmark.
5 comments
AuthorsOpenReview ↗

Response to R. GMxd (Page 1/2)

Thank you for your insightful comments! Below are our detailed responses.

W1: The principal metric does not establish decision quality, despite the paper’s decision-making framing. Appendix F.1 provides a particularly clear example: both the true optimizer and the empirical optimizer choose the same optimal order quantity, (Q=91), yet the empirical solution receives a 7.37% error because its estimated profit differs from the true expected profit. Thus, a genuinely optimal operational decision can be counted as an inaccurate answer. The results support claims about optimal-value estimation, but not directly about poor decisions, infeasibility, or decision regret.

Q2: The benchmark evaluates the error in the reported optimal objective value rather than the quality of the resulting decision. As illustrated by the paper’s own example, the empirical and true problems can produce the same optimal decision while still receiving a sizable objective-value error. Could the authors report decision-level metrics, such as decision regret under the true distribution or the fraction of cases in which the inferred and true optimal decisions agree?

Thank you for highlighting this important distinction. We agree that objective-value error does not equal to decision quality, as acknowledged in our Limitations section. In some given cases, such as the example in Appendix F.1, as you point out, the empirical optimizer selects the same optimal decision but still incurs 7.37% objective-value error due to finite-sample estimation.

The main reason we adopt objective-value-based evaluation is to provide a consistent, automatically verifiable, and scalable metric across different OR problems, as some prior works also did [1]. Decisions can take different forms across OR problems, ranging from simple quantities to multi-stage decisions and complex policies. Moreover, optimal decisions may not be unique, making it impractical to enumerate all optimal solutions for direct decision matching. Finding the ground-truth decisions requires substantial human involvement in problem-specific decision extraction, feasibility checking, and verification, thus limiting the scalability of benchmark construction. Therefore, we use objective-value error as our primary metric.

We recognize your concerns about this limitation, so we conducted an additional decision-level analysis on the paired NL2Code and Workflow tasks under the Python-only setting with 100 observations. Specifically, we used an LLM judge (GPT-6.1 Sol) to extract the decisions from the original agents' execution trajectories and grouped the paired tasks based on whether their decisions were consistent. We then compared the original Workflow objective-value errors between these two groups. Given that the strongest models achieve near-perfect performance on NL2Code tasks, we expect Workflow decisions that differ from their NL2Code counterparts to be associated with higher objective-value errors.

As shown in Table A, objective-value error is consistently around 2–3 times higher when the Workflow decision differs from its NL2Code counterpart, where the strongest models achieve near-perfect performance. While decision consistency is not equivalent to decision regret, these results provide additional evidence that the observed observation-to-decision gap is associated not only with objective-value estimation errors, but also with differences in the decisions made by the models. This suggests that our current metric can provide reference to the decision quality difference.

Table A. Original Workflow objective-value errors on paired L1 tasks, grouped by whether the LLM-judge-extracted decisions agree with those of their NL2Code counterparts.

ModelWorkflow Err. (same decision w/ NL2Code)Workflow Err. (different decision w/ NL2Code)
GPT-4o14.34%33.57%
GPT-5.25.58%15.92%
GPT-5.6 Luna3.59%11.48%
GPT-5.6 Terra3.85%10.36%
GPT-5.6 Sol3.77%10.79%
GPT-6 Astra3.90%10.20%

We will include the complete additional decision-level results in the revised manuscript and further clarify the distinction between objective-value error and decision quality. We really appreciate you pointing this out.

AuthorsOpenReview ↗

Response to R. GMxd (Page 2/2)

W2 & Q1: The paper’s “observation-to-decision” framing is somewhat broader than the actual task. In the Workflow setting, the optimization structure (the decision problem, objective, and constraints) is already specified in natural language; the agent mainly needs to estimate hidden stochastic parameters from 100 observations and then solve the resulting optimization problem. Thus, the benchmark does not test whether an agent can infer the underlying decision problem itself from operational data.

Thank you for this great point. The main scope of our ORWorkerBench is to evaluate observation-driven stochastic optimization within a specified decision problem. The operational objective and constraints are supplied in natural language; agents must infer the hidden stochastic quantities and use them to formulate and solve the optimization problem. We do not evaluate discovery of the objective or constraint structure from raw data. Under this setting, agents identify and estimate decision-relevant stochastic parameters from finite observations before formulating and solving the optimization model.

Regarding the concern about inferring the underlying decision problem from operational data, we would like to share our thoughts about the design choice. While the decision problem, objective, and constraints are provided in natural language, translating these descriptions into appropriate OR formulations and executable solver code remains a challenging task that has received substantial attention from the research community. Building upon the traditional NL2Code setting, we <u>extends OR problem solving to an observation-driven operational setting, where decision-relevant stochastic parameters are not explicitly given but must instead be inferred from historical observations</u>. Therefore, we believe the work represents a meaningful and challenging step beyond the traditional NL2Code paradigm.

We totally agree with your insights on the research direction. We will clarify the scope of "from data to decisions" in the Introduction section of the camera-ready version and discuss problem discovery from real-world document or messy operational data as a valuable future extension to explore.

Q3: The strongest models perform very close to the same-data estimate-then-optimize reference. How should we interpret the remaining approximately 7% error as an LLM limitation rather than ordinary finite-sample estimation error? Can the authors provide stronger statistical estimation baselines or an oracle/statistical benchmark that better quantifies the achievable performance given only 100 observations?

Q4: All Workflow tasks use exactly 100 observations. How does the observation-to-decision gap change with sample size (e.g., 25, 50, 200, or 500 observations)? Without such a comparison, it is difficult to determine whether the reported gap primarily reflects limitations of LLM reasoning or simply the particular finite-data regime chosen by the benchmark.

Thank you for this point. We view the same-data reference as a comparator for the effect of finite-sample variability on objective-value estimation**.** Thus, the strongest models’ proximity to this reference suggests that their remaining approximately 7% error should not be attributed entirely to LLM limitations. Our additional experiments, as shown in Table B, increasing the number of observations from 100 to 400 consistently reduces error across five models. This sweep covers Regular tasks only and reuses historical runs with 100 observations. Interestingly, while the best-performing model, Astra, outperforms the same-data reference baseline with 100 observations, it falls behind the reference at both 200 and 400 observations, despite the consistent error reduction brought by larger observation budgets.

Table B. Workflow objective-value error with 100, 200, and 400 observations, on Regular Split.

Model100 obs.200 obs.400 obs.
GPT-4o19.3518.1318.11
GPT-5.26.584.503.26
GPT-5.6 Luna6.334.593.69
GPT-5.6 Terra6.153.803.07
GPT-5.6 Sol5.763.642.77
GPT-6 Astra5.763.652.83
Same-data reference5.823.552.79

We will clarify that this reference contextualizes finite-data effects; it does not establish the best achievable statistical performance or precisely decompose all error sources. Stronger statistical estimators remain a valuable direction for further evaluation.

Reference:

[1] Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, and Yinyu Ye. Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling. Advances in Neural Information Processing Systems (NeurIPS 2025), Volume 38, 2025.

Thank you for the additional experiments and clarifications. The sample-size sweep and the more careful interpretation of finite-sample error meaningfully address my concerns.

My main remaining concern is decision quality. The decision-consistency analysis is useful, but disagreement with the paired NL2Code decision does not necessarily imply higher regret, particularly when multiple optimal solutions exist. Non-uniqueness does not prevent evaluating a returned decision under the true distribution. Could you report feasibility and decision regret on at least a subset of tasks, ideally alongside the same-data reference?

Reviewer_PegSOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Typical optimization benchmarks assume that the quantities needed to formulate and solve the problem are already well-specified. In real life, however, some of these quantities describe uncertainty and must be estimated from observational data. To study this setting, the authors introduce ORWorkerBench, which compares problems with explicitly provided stochastic parameters against matched problems where those parameters must be estimated from finite observations. They additionally use neuron-guided synthesis to select problems with more diverse activation patterns.

They evaluate their benchmark using several frontier models and find there is a significant gap between the two scenarios, with models often saturating well-specified problems but performing poorly on infer-only problems, even though the underlying problem structure is the same. They also find that strong models struggle to beat simple baselines (e.g. statistically estimating the quantities from the input observations), highlighting the need for models to make better decision making with finite observations.

Critical strengths

This paper is practically motivated and provides new insights on how LLMs solve optimization problems. I believe it moves us closer to studying how LLMs can be deployed to solve optimization problems in real operational settings (e.g. estimating uncertain inputs from historical observations). The proposed methods to generate novel and diverse optimization problems is also interesting: they run sample problems through a small model and observe neuron activation diversity, and select examples which maximize the set cover of neurons

Critical weaknesses

There are some limitations, like only generating 441 Workflow tasks, and only 147 of such tasks having a matching number of NL2Code tasks. I would also like to see a sweep varying the number of input observations - does the model improve further at stochastic estimation with more inputs?

Questions for authors

If you increase the number of input observations per task, how does that change the models' performance?

1 comment
AuthorsOpenReview ↗

Response to R. PegS

Thank you for your encouraging comments! Below are our detailed responses.

W1.1 & Q1: I would also like to see a sweep varying the number of input observations - does the model improve further at stochastic estimation with more inputs?

Thank you for the insightful comment. To further support our benchmark, we conducted an additional experiments with 100, 200, and 400 observations across the same 336 Regular tasks, averaging four Python-only runs per model and condition. Mean capped objective-value errors (%) are as follows.

Table B. Workflow objective-value error with 100, 200, and 400 observations, on Regular Split.

Model100 obs.200 obs.400 obs.
GPT-4o19.3518.1318.11
GPT-5.26.584.503.26
GPT-5.6 Luna6.334.593.69
GPT-5.6 Terra6.153.803.07
GPT-5.6 Sol5.763.642.77
GPT-6 Astra5.763.652.83
Same-data reference5.823.552.79

Increasing the number of observations from 100 to 400 consistently reduces error across five models. This sweep covers Regular tasks only and reuses historical runs with 100 observations. Interestingly, while the best-performing model, Astra, outperforms the same-data reference baseline with 100 observations, it falls behind the reference at both 200 and 400 observations, despite the consistent error reduction brought by larger observation budgets.

W1.2: There are some limitations, like only generating 441 Workflow tasks, and only 147 of such tasks having a matching number of NL2Code tasks.

We understand there could be some concern about the scale of the benchmark. To help addressing the concern, we first list the scale of our precedent work below, from large to small.

BenchmarkDataset SizeNotes
NL4Opt (Ramamonjison et al., 2022)1,101289 test tasks
OptiVerse (Zhang et al., 2026)1,000Multiple optimization paradigms
MAMO (Huang et al., 2025b)863652 EasyLP + 211 ComplexLP
OptiBench (Yang et al., 2025)605Optimization modeling tasks
ORWorkerBench (Ours)588147 NL2Code + 441 Workflow
NLP4LP (AhmadiTeshnizi et al., 2025)354287 Easy + 67 Hard
OR-Space (Zhou et al., 2026)300100 topologies × 3 task views
FrontierOR (Kong et al., 2026a)180Literature-grounded OR tasks
ORAgentBench (Li et al., 2026)107Executable agent tasks
IndustryOR (Huang et al., 2025a)100Industrial OR problems

Therefore, we would like to clarify that 441 Workflow tasks is comparable to the scale of existing executable OR agent benchmarks, such as FrontierOR (180), OR-Space (300), and ORAgentBench (107). While larger benchmarks such as OptiVerse (1000) exist, they primarily evaluate optimization problems with specified inputs rather than observation-driven workflows.

Reference:

See paper’s reference.

Reviewer_HfcTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ORWorkerBench, a benchmark designed to evaluate the ability of large language models (LLMs) to infer unknown stochastic parameters from limited historical observations and subsequently solve optimization problems. For each underlying optimization task, the benchmark provides two evaluation settings, an "NL2Code" condition where parameters are explicitly given and a "Workflow" condition where parameters must be inferred from 100 observations. The benchmark comprises 21 human-verified executable protocols, 147 NL2Code instances, and 441 Workflow instances. The authors employ a neuron-activation-guided approach for task construction and provide a same-data reference pipeline that estimates parameters from the observations and passes them to a verified optimization solver.

Critical strengths
  1. By establishing both parameters-known and observation-driven conditions for the same optimization problem, the benchmark enables a clearer analysis of how LLM performance changes when stochastic parameters must be inferred rather than explicitly provided.

  2. The inclusion of a reference solving pipeline based on the same observational data provides a useful baseline for assessing how much error may arise from finite-sample parameter estimation rather than the model's own optimization workflow.

  3. The construction process, which integrates executable reference solvers with manual quality verification, enhances the reliability and reproducibility of the benchmark.

  4. The use of neuron-activation-guided task selection increases activation-pattern diversity across multiple models and offers a useful strategy for reducing redundancy among benchmark tasks.

  5. The experiments reveal a substantial performance gap between parameters-known and observation-driven optimization tasks, highlighting a limitation of existing evaluations that primarily focus on optimization problems with explicitly specified parameters.

Critical weaknesses
  1. The evaluation relies primarily on the prediction error of the optimal objective value, without directly assessing the quality or feasibility of the actual decisions made. Consequently, there is a disconnect between the evaluation metrics and the paper's stated focus on the model's ability to derive decisions from data.

  2. While the paired evaluation design is a key contribution, existing research already covers stochastic optimization benchmarks [1] and neuron-activation-based data selection methods [2]. Therefore, the methodological novelty of the proposed framework is somewhat limited.

[1] OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving. ACL 2026 findings . [2] Neuron-Aware Active Few-Shot Learning for LLMs. ACL 2026.

Questions for authors
  1. Given that substantial relative objective error can arise even when the optimal decision is selected (Appendix F.1), why does ORWorkerBench evaluate scalar objective values rather than decision regret? Would the main conclusions about the observation-to-decision gap still hold under a decision-based metric?
  2. Beyond activating diversity, can neuron-guided synthesis further enhance mathematical diversity or model discriminative capability?
2 comments
AuthorsOpenReview ↗

Response to R.HfcT (Page 1/2)

Thank you for taking the time to review our paper and provide thoughtful suggestions! We provide more detailed responses to your points below.

W1: The evaluation relies primarily on the prediction error of the optimal objective value, without directly assessing the quality or feasibility of the actual decisions made. Consequently, there is a disconnect between the evaluation metrics and the paper's stated focus on the model's ability to derive decisions from data.

Q1: Given that substantial relative objective error can arise even when the optimal decision is selected (Appendix F.1), why does ORWorkerBench evaluate scalar objective values rather than decision regret? Would the main conclusions about the observation-to-decision gap still hold under a decision-based metric?

We appreciate you bringing up this important point. We agree that objective-value error does not equal to decision quality, as acknowledged in our Limitations section. However, we adopt this metric to provide a consistent and solver-verifiable evaluation across different OR problems, following prior work [3]. Moreover, similar to other open-ended tasks, since optimal decisions may not be unique, directly matching a model's decision against a single reference decision could underestimate its performance.

To further address this concern, we conducted an additional decision-level analysis using GPT-6 Astra as an LLM judge to compare the decisions made in paired NL2Code and Workflow tasks.

We find that the decisions differ in approximately half of the paired tasks, and the objective-value error is consistently 2–3 times higher when the decisions differ, as shown in Table A. This suggests that the observed gap is not solely caused by inaccurate objective-value estimation, but is also associated with differences in the decisions made by the models. While decision consistency is not equivalent to decision regret, these results provide additional evidence supporting our findings of observation-to-decision gap.

It is also worth note that, to account for errors that may arise from finite observations even when the estimated optimization problem is solved correctly, we introduce a same-data empirical reference (Sec. 3.1). This reference uses exactly the same observations as the evaluated agents and passes the estimated parameters to a verified solver, allowing us to measure the additional error introduced by the end-to-end workflow beyond a fixed estimate-then-optimize pipeline.

We will include this additional decision-level analysis and the LLM judge reliability evaluation (of this small experiment) in the revised manuscript, and further clarify the distinction between objective-value error and decision quality.

Table A. Workflow objective-value error on L1 tasks where the model's decision is the same as or different from its decision on the paired NL2Code task.

ModelErr. (same)Err. (different)
GPT-4o14.34%33.57%
GPT-5.25.58%15.92%
GPT-5.6 Luna3.60%11.48%
GPT-5.6 Terra3.86%10.36%
GPT-5.6 Sol3.77%10.79%
GPT-6 Astra3.90%10.20%
AuthorsOpenReview ↗

Response to R.HfcT (Page 2/2)

W2: While the paired evaluation design is a key contribution, existing research already covers stochastic optimization benchmarks [1] and neuron-activation-based data selection methods [2]. Therefore, the methodological novelty of the proposed framework is somewhat limited.

Thank you for recognizing our paired evaluation design. We agree that stochastic optimization benchmarks [1] and neuron-activation-based selection methods [2] have both been explored in previous work, and we do not claim either of them as a standalone contribution.

Meanwhile, we would like to emphasize that the contribution of ORWorkerBench goes beyond simply including stochastic optimization problems. ORWorkerBench pushes the evaluation of OR agents beyond the traditional NL2Code paradigm to the Workflow level, which requires agents not only to understand how to formulate and solve OR problems, but also to analyze critical modeling parameters from data.

Regarding [2], while neuron-based data selection methods have been proposed in previous work, <u>the use of neuron activation signals for benchmark synthesis and diversity analysis remains relatively underexplored</u>. Rather than proposing a new selection algorithm, we apply these signals to benchmark construction and empirically demonstrate that neuron-guided selection produces more diverse tasks than embedding-based selection, with improvements generalizing to held-out models. We will further clarify these distinctions in Sec. 2 and our contribution list.

Q2: Beyond activating diversity, can neuron-guided synthesis further enhance mathematical diversity or model discriminative capability?

Thank you for the insightful question. To provide more evidence for it, we ran an additional correlational experiment. It provides evidence that neuron activation patterns capture mathematical structural diversity better than hidden-state or embedding-model representations.

Setup. We use NL4Opt, whose problems carry explicit annotations of their mathematical formulation. For each of the 1,098 annotated problems, we represent its mathematical structure by a sparse feature vector read directly from the annotation, with no model involved: (1) the number of decision variables, (2) the number of constraints, (3)the count of each of the seven annotated constraint types (sum, upper bound, lower bound, linear, ratio, and the two variable-comparison types), and (4) the objective direction.

We generate one reasoning rollout per problem and encode the same rollout in three ways:

  • Activation: the mean MLP neuron activation at the middle layer.
  • Hidden states: the mean final-layer hidden state.
  • Embedding model: the sentence embedding (SimCSE, or Qwen3-Embedding-0.6B) of the rollout text.

We sample 50 of the 1,098 problems at random, 1,000 times. For each subset, we compute the Vendi Score of each representation and of the structural features. We report the Spearman correlation ρ, across subsets, between the diversity of each representation and structural diversity, averaged over 4 sampling seeds.

Results.

Rollout Modelρ (activation)ρ (hidden states)ρ (Embedding-SimCSE)ρ (Embedding-Qwen3-Embedding-0.6B)
DeepSeek-R1-Distill-Llama-8B0.3750.2200.1960.291
Qwen3.5-2B0.3070.2540.1290.268

On both models, activation diversity is the most strongly correlated with structural diversity. The gap between activations and each of the other two representations is positive in all four seeds. We will add this experiment and its findings to the appendix as an additional support.

[3] Huang, Chenyu, et al. "ORLM: A customizable framework in training large models for automated optimization modeling." Operations Research 73.6 (2025): 2986-3009.

Comments on the paper

2 comments

Dear Reviewers and ACs,

With the discussion phase near its conclusion, we sincerely thank you all for your insightful comments and valuable suggestions. We are encouraged that all reviewers recommend acceptance, and we are glad that Reviewer has shared that our additional decision-level evaluation substantially addressed their main concern and raised the rating. Below, we provide a summary of the contributions recognized, additional experiments during the discussion phase, and our detailed plan of revisions.

Contributions of the Paper:

  • An observation-to-decision optimization benchmark. We introduce ORWorkerBench, which evaluates whether LLM agents can infer decision-relevant stochastic quantities from finite observations before solving the downstream optimization problem. This setting is practically motivated (R PegS). Its 21 human-verified executable protocols, each with a schema, a stochastic simulator, and a verified solver, are substantive benchmark assets (R GMxd), and the construction process enhances the reliability and reproducibility of the benchmark (R HfcT).
  • Matched observation-to-decision evaluation. Each decision problem is evaluated in paired NL2Code and Workflow settings, which enables a clearer analysis of how performance changes when stochastic parameters must be inferred (R HfcT). Together with the same-data reference, a useful baseline (R HfcT) that is particularly valuable (R GMxd), this design separates finite-sample estimation error from the error introduced by the agent, making the benchmark diagnostically useful (R GMxd).
  • Neuron-guided benchmark synthesis. We use activation patterns to guide prototype discovery and instance expansion, which offers a useful strategy for reducing redundancy among benchmark tasks (R HfcT) and is interesting (R PegS). Under controlled candidate batches and generation budgets, it produces more diverse activation patterns than embedding-based selection across four held-out models (R GMxd).
  • Empirical study of frontier LLMs. Our evaluation of eight LLMs reveals a substantial observation-to-decision gap: the strongest models nearly saturate parameter-specified tasks, yet remain unsatisfying once the stochastic quantities must be inferred from observations, and neither stronger models nor specialized tools close this gap. These findings provide new insights on how LLMs solve optimization problems (R PegS), highlighting a limitation of existing evaluations (R HfcT) that focus on explicitly specified problems. Our paper is also clear and easy to follow (R PegS, GMxd).

Additional Experimental Evidence Summary:

During the discussion, we added additional experiments and analysis to further address questions from our reviewers:

  • The gap extends to decision quality (RQ1). We evaluated the decisions extracted from agent trajectories under the true underlying distribution on all 147 paired L1 tasks, alongside the same-data reference. Any feasible optimal decision receives zero error, so this evaluation accommodates multiple optimal solutions. The results confirm that the observation-to-decision gap is reflected in actual decision quality, not only in objective-value estimation. Consistently, Workflow objective-value error is 2–3 times higher when a model's decision differs from its paired NL2Code decision.
  • The findings hold beyond the 100-observation budget (RQ2). We varied the observation budget from 100 to 200 and 400 on the 336 Regular tasks. Error decreases consistently as observations grow, while the strongest models continue to track the same-data reference: GPT-6 Astra slightly outperforms the reference with 100 observations but falls behind it with 200 and 400 observations. Under the decision-level evaluation, five of the six proprietary models also incur higher mean decision error than the same-data reference. Together, these results show that finite-sample estimation explains a substantial part of the gap, while current LLM agents still do not reliably go beyond a simple estimate-then-optimize pipeline.
  • Neuron activations reflect mathematical structure (neuron-guided synthesis). On 1,098 NL4Opt problems with annotated formulations, activation diversity correlates more strongly with mathematical structural diversity than hidden-state and embedding-model representations, across two rollout models and all four sampling seeds. This connects activation diversity to problem structure measured independently of any model.

All additional results and clarifications will be incorporated into the revised manuscript according to the above plan. Thank you once again for your efforts in reviewing our work. We hope our responses have adequately addressed your concerns.

Warm Regards, The Authors of Submission 46669

A Summary of Responses and Revisions:

Responses and Revisions:

  • For Reviewer HfcT's concerns:
    • On decision quality (W1, Q1), we provided the decision-consistency analysis and the decision-level evaluation under the true distribution described above, and clarified how the same-data reference isolates finite-sample error from the error introduced by the end-to-end workflow.
    • On novelty (W2), we clarified that we do not claim stochastic optimization benchmarks or neuron-based selection as standalone contributions. Our contribution lies in extending OR evaluation from the NL2Code paradigm to the Workflow level under matched conditions, and in applying neuron activation signals to benchmark synthesis and diversity analysis. We will sharpen these distinctions in Sec. 2 and the contribution list.
    • On mathematical diversity (Q2), we provided the NL4Opt structural-diversity experiment described above.
  • For Reviewer PegS's concerns:
    • On the number of observations (W1, Q1), we provided the 100/200/400-observation sweep described above.
    • On benchmark scale (W1), we showed that the 441 Workflow tasks are comparable to or larger than existing executable OR agent benchmarks, such as FrontierOR (180), OR-Space (300), and ORAgentBench (107), while ORWorkerBench is the only benchmark covering both stated and latent stochastic parameters (Table 1).
  • For Reviewer GMxd's concerns:
    • On decision quality (W1, Q2), in addition to the decision-consistency analysis, we followed the reviewer's suggestion and evaluated decision error and agreement with the oracle decision under the true distribution on all 147 paired L1 tasks, alongside the same-data reference. As the reviewer requested, we will clarify the reliability of decision extraction, the treatment of infeasible decisions, and error normalization in the revision.
    • On scope (W2, Q1), we clarified that ORWorkerBench evaluates observation-driven stochastic optimization within a specified decision problem. We will make this explicit in the Introduction and discuss problem discovery from raw operational data as future work.
    • On finite-sample error (Q3, Q4), we provided the observation-budget sweep and clarified that the same-data reference contextualizes finite-sample effects rather than serving as a statistical lower bound.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.