acceptodds
Under review as a conference paper at ICLR 2027

OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

13 views

Abstract

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks commonly begin with self-contained problem descriptions or predefined optimization tasks, leaving the work of translating original business materials into professionally validated solutions underrepresented. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios drawn from paid OR engagements totaling $3.68 million in project value. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To evaluate information acquisition throughout optimization, we further introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero credit to infeasible deliveries. Across 11 frontier LLMs evaluated with coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest system achieves 83.87% strict feasibility and a DAQ of 0.84 against the expert reference of 1. Client information acquisition extends beyond initial clarification into solution development and validation, with some later stages exhibiting higher confirmed issue-resolution proportions. Trajectory analysis identifies failures spanning formulation and implementation, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Operations Research, Agentic AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9AimOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OR-License, a benchmark for evaluating whether LLM agents can perform end-to-end operations research (OR) tasks using original client materials rather than fully specified optimization problems. The benchmark contains 31 instances across 10 industrial scenarios drawn from paid OR projects, with client-accepted expert solutions serving as references. It also introduces ORSim, a client simulator that allows agents to request additional business information throughout the optimization process without receiving OR expertise. Agent submissions are evaluated using hidden feasibility checks and expert-relative objective quality, including a feasibility-gated metric called Deployment-Adjusted Quality (DAQ). The authors evaluate 11 LLMs using coding-agent frameworks and examine their information-seeking behavior, research processes, and failure patterns. The results show that some models outperform the expert references in objective quality on feasible instances, but none matches the expert reference in DAQ. The best-performing system achieves 83.87% strict feasibility and 83.58% DAQ. The paper also reports a separate reliability evaluation of ORSim and analyzes common difficulties in translating business requirements into feasible optimization solutions.

Critical strengths

The main strength of this paper is the construction of a benchmark from real industrial OR projects. OR-License contains 31 instances across 10 industrial scenarios, with original client requirements, input data, and expert solutions accepted by clients before benchmark construction. Unlike benchmarks based on predefined optimization problems, it evaluates whether agents can interpret business requirements and produce executable solutions. This addresses an important gap in existing OR benchmarks and provides a valuable resource for studying LLM agents in professional settings.

Another strength is the design of ORSim, which allows agents to acquire business information throughout the optimization process while restricting access to OR expertise. The separation of fact selection, deterministic authorization, and response generation makes the interaction more controlled and auditable. The authors also evaluate the simulator's reliability using 204 expert-annotated queries with repeated trials. Together with hidden feasibility checks and expert-relative objective evaluation, this provides a reasonably well-designed framework for assessing end-to-end OR performance.

Critical weaknesses

My concern is that the experiments do not directly answer RQ2 as stated. The paper aims to examine how allowing agents to acquire client information throughout optimization affects solution feasibility and objective quality. However, Section 4.2 and Figure 5 mainly report when agents ask questions and the proportion of questions associated with confirmed issue resolution. There is no controlled comparison between agents that can ask questions only during initial clarification and agents that can continue asking throughout the optimization process. The reliability experiments in Appendix C provide useful evidence about ORSim's response quality, but they do not establish whether continued access to client information improves final optimization performance. The current results support the observation that information acquisition extends beyond initial clarification, but they cannot establish its effect on feasibility or DAQ.

Questions for authors

The paper identifies information acquisition throughout optimization as a central contribution, and RQ2 aims to examine its effect on feasibility and objective quality. However, Section 4.2 mainly reports the timing of client questions and their confirmed issue-resolution rates. Could the authors clarify whether RQ2 is intended to characterize information-seeking behavior or to establish the benefit of continued access to ORSim? If the latter, is there evidence comparing agents that can ask questions only during initial clarification with agents that can continue asking throughout optimization? This distinction would affect how strongly the experimental results support the claimed contribution.

Appendix G.2 describes expert reviews of the consistency between the original requirements, agent-facing materials, and hidden evaluators. Could the authors clarify how they verified that all business requirements enforced by the hidden evaluators could be recovered from the initial materials, input data, or authorized ORSim responses? In particular, was this information coverage systematically checked for all 31 instances? This is important for determining whether failures primarily reflect limitations in the agents' ability to understand and formulate OR problems, rather than information that was not accessible during evaluation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9Aim (Part 1/2)

We sincerely thank the reviewer for recognizing the novelty and importance of OR-LICENSE, particularly its dataset contribution based on real industrial OR projects, and the design of ORSim. We appreciate your insightful comments on the distinction between characterizing agents' information-seeking behavior and establishing the benefits of sustained client interaction. We address your concerns through two complementary evaluations: (1) a comparison of three client-information interaction settings to examine their effects on final solution performance, and (2) a systematic information-coverage validation across all 31 instances to verify that the requirements enforced by hidden evaluators are recoverable from the information available to agents. We detail these additional evaluations and their findings below.


W1&Q1: Comparisons across different information-interaction settings

Thank you for this suggestion. RQ2 aims to evaluate how continued access to client information affects final optimization performance. We agree that a direct comparison between initial clarification only and continuous interaction is needed to assess the effect of retaining access to ORSim after the initial phase.

To this end, we compared the following two settings across the same 11 models and all 31 tasks, with five runs per model–task pair in each setting:

  • Initial clarification only: Agents may inspect the supplied materials and query ORSim during initial clarification. Client querying is permanently disabled immediately before the agent first writes or modifies code implementing an optimization model or solution algorithm, or first invokes a solver or optimization routine, whichever occurs first. Reading files and inspecting their schema or summary statistics do not trigger this cutoff.
  • Continuous interaction: Agents retain access to ORSim throughout formulation, solving, validation, and revision, allowing further questions as new information needs arise.

We evaluated final submissions using strict feasibility and DAQ. DAQ incorporates objective quality relative to the expert reference and assigns zero credit to infeasible submissions. The results are shown below, with the final row reporting the equally weighted mean across the 11 models.

ModelInitial only: Feasibility (%)Initial only: DAQContinuous: Feasibility (%)Continuous: DAQ
Claude Opus 535.48 ± 9.1237.61 ± 9.1943.87 ± 38.9544.07 ± 39.08
DeepSeek-V4.1-Flash47.74 ± 11.5047.17 ± 10.6552.90 ± 30.6552.21 ± 30.12
Doubao-Seed-2.1-Pro32.26 ± 6.0331.39 ± 5.4162.58 ± 3.6862.55 ± 2.61
Gemini 3.8 Flash60.00 ± 2.8962.31 ± 3.5668.39 ± 5.7768.60 ± 3.96
GLM-5.3-Flash67.74 ± 5.1067.79 ± 4.6561.94 ± 5.7761.49 ± 6.74
GPT-5.6 Sol61.29 ± 9.6865.27 ± 10.2369.03 ± 13.9971.58 ± 14.75
Grok 4.681.94 ± 5.4081.41 ± 6.2883.23 ± 7.7082.40 ± 9.78
Hy340.00 ± 5.4039.14 ± 4.7543.23 ± 7.0741.41 ± 5.75
Hy4 preview46.45 ± 7.4345.96 ± 7.9751.61 ± 17.9651.01 ± 19.71
Muse Spark 1.356.77 ± 2.8956.69 ± 3.5470.97 ± 5.5969.56 ± 4.17
Qwen3.8-Max48.39 ± 4.5650.28 ± 4.5256.13 ± 13.6156.48 ± 15.51
Equal-weight mean52.5553.1860.3560.12

Continuous interaction increases average feasibility from 52.55% to 60.35%, a gain of 7.80 percentage points, and average DAQ from 53.18 to 60.12, a gain of 6.94 points. Ten of the eleven models achieve higher mean feasibility and DAQ. These results show that, in this evaluation, retaining access to client information after initial clarification improves final delivery performance for the vast majority of evaluated models and yields a positive average gain.

AuthorsOpenReview ↗

Response to Reviewer 9Aim (Part 2/2)

Model-level results also reveal variation: GLM-5.3-Flash shows lower mean feasibility and DAQ under continuous interaction, and the relative rankings of models also change. When ranked by mean DAQ, eight of the eleven models change position, with Doubao rising from 11th to 5th and Muse from 5th to 3rd. Together, these results show that the gains from continued access to client information vary across models, and that information access conditions affect assessments of their relative performance.

We have incorporated this comparison into Section 4.2 of the revised paper, using changes in final feasibility and DAQ as the primary evidence for answering RQ2. Figure 5 provides a behavioral analysis of when agents seek information and the associated evidence of issue resolution, while Appendix C continues to report the reliability of ORSim’s responses. Together, these analyses present evidence on final performance, the interaction process, and simulator reliability.


Q2: Systematic Information-Coverage Validation

We thank the reviewer for raising this important question. During benchmark construction, we systematically checked information coverage for all 31 instances. Specifically, for each instance, the responsible domain expert checked the business requirements enforced by the hidden evaluator individually and verified that the information needed for each requirement could be obtained through the same channels available to evaluated agents: the initial business brief, input data, and direct interaction with ORSim. For requirements requiring client clarification, the experts also queried ORSim themselves to confirm that the relevant facts could be obtained through authorized responses.

Furthermore, we recruited five additional OR practitioners with at least two years of industry experience who were not involved in benchmark construction to independently validate information accessibility. Each expert personally interacted with ORSim acting as the client and assessed whether the business requirements enforced by the hidden evaluators could be recovered from the initial materials, input data, and authorized ORSim responses.

Evaluation dimensionRating (mean ± SD) ↑Pairwise Exact Agreement ↑95% Bootstrap CIVerified information accessibility ↑
Accessibility of business requirements4.99 ± 0.1198.06%[94.19%, 100.00%]100%

The Verified information accessibility measures the proportion of predefined, authorized information items that experts successfully acquired through direct interaction with ORSim, aggregated across expert–instance combinations. A value of 100% requires both complete acquisition of the audited information items and expert confirmation that the information needed for every hidden evaluation requirement is accessible. We have added the systematic information-coverage validation across all 31 instances to Appendix G.2 and the independent expert assessment of information accessibility to Appendix C.5 of the revised manuscript.

Together, these results show that information accessibility was systematically verified during task construction for all 31 instances and independently re-examined by five practitioners who had not participated in benchmark construction. Both checks provide evidence that the business requirements enforced by the hidden evaluators can be recovered through the information channels available to agents, including requirements that require active client questioning. However, information being accessible does not guarantee that agents will successfully acquire and use it. Agents must still identify information gaps, ask effective questions, and translate the acquired business requirements into correct optimization formulations and executable solutions. These are central capabilities that OR-License aims to evaluate.

Dear Reviewer,

We hope you are well. We would like to express our thanks once again for your helpful comments. As the discussion period draws to a close, we wanted to follow up and see whether our responses have satisfactorily addressed the points you raised. Please feel free to let us know if any aspect would benefit from further clarification. We appreciate your insights and would be glad to address any additional feedback.

Thank you again for your thoughtful review and the time you have dedicated to our paper.

Reviewer_Aui7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

It proposes an end-to-end LLM agent evaluation framework based on industrial operations research tasks. Unlike existing benchmarks that mainly rely on fully specified optimization problems with complete information, this framework aims to evaluate agents' capabilities in information gathering, problem abstraction, mathematical modeling, optimization solving, and final delivery in real business environments. The dataset is based on 31 real-world cases from 10 industrial scenarios. The framework uses the ORSim simulator to simulate the interaction process and collect client business knowledge; it evaluates agents in terms of feasibility and quality using constraint verifiers and expert solutions; and it analyzes execution trajectories through HCAEU. The experiments evaluate 11 LLM agents. Some models perform better than expert models on the quality metric, but their overall DAQ scores remain below the expert level. Meanwhile, trajectory analysis is used to further analyze the stages where agents fail.

Critical strengths
  1. Strong real-world relevance, taking into account practical needs in real-world scenarios and offering significant practical value.
  2. The Benchmark framework is comprehensively designed, taking into account the collection and preprocessing of real project materials, simulated client interactions, solution verification, and expert-reference evaluation. It also defines an evaluation metric that jointly considers feasibility and optimization quality, as well as HCAEU for analyzing execution trajectories.
Critical weaknesses
  1. The real-world authenticity and distinctive evaluation value of the benchmark have not been sufficiently validated. Although the tasks originate from real projects, after being organized and structured, it remains unclear whether these processes preserve the ambiguity, incomplete information, and dynamic interactions of real business scenarios. For example, although ORSim provides controlled information interactions, it is unclear whether it can adequately simulate issues such as ambiguous expressions, missing information, and communication difficulties encountered with real clients.
  2. The main model comparison experiments rely on the Claude Code harness, while the Codex comparison provided in the appendix is not a strictly paired experiment, making it difficult to effectively rule out the influence of differences in execution frameworks on model rankings. The actual computational resources and reasoning configurations also differ across models. Therefore, the experimental results are more appropriately interpreted as the performance of agent systems under specific execution configurations, rather than as a general ranking of the models' inherent professional capabilities.
Questions for authors
  1. In the design of DAQ, quality scores can compensate for failed tasks. Does this align with the research goal of reliable expert-level delivery? In theory, a small number of exceptionally high-quality solutions can offset infeasible results on other instances. If the paper aims to measure whether an agent is qualified to perform professional operations research work, should it set an independent minimum requirement for strict feasibility, rather than relying solely on a compensatory aggregate score?
  2. Different industrial tasks may have different levels of importance and difficulty. Could the average score mask differences in failures across tasks?
  3. Is it reasonable for the ORSim system to limit each response to only three facts? Could this limitation affect the agent's information-gathering efficiency and final task performance?
14 comments
AuthorsOpenReview ↗

Response to Reviewer Aui7 (Part 1/10)

We sincerely appreciate your recognition of the practical value of our benchmark, its comprehensive end-to-end design spanning real-world data curation, client interaction, solution verification, expert-reference evaluation, and our evaluation metrics design. We are also grateful for your positive assessment of the paper's presentation. In response to your concerns, we have conducted extensive additional experiments and analyses to further validate the benchmark's real-world fidelity, establish a more rigorous comparison between execution harnesses, clarify the role of DAQ, and examine the impact of ORSim's per-response fact-disclosure limit. We address each of these concerns with additional evidence and clarifications below.


W1: Real-World Scenario Representativeness and Distinctive Evaluation Value

We thank the reviewer for raising the important question of whether our task construction and client simulation preserve the characteristics of real-world business interactions. We address this concern through three complementary forms of evidence: expert-led task construction, information-recovery analysis, and an independent expert assessment of ORSim's interaction fidelity. We further examine the distinctive evaluation value of OR-LICENSE by comparing its results with the Epoch Capabilities Index (ECI) [1] and evaluating the effects of different information-access conditions.

Expert-led construction preserves original business requirements and information boundaries.

The task curation and client-fact annotation were performed by domain experts who contributed the original OR projects, as described in Section 3.2 and Appendices G.1–G.2. Specifically, these experts:

  • De-identified the original requirements, input data, and deliverable materials while preserving their original content, terminology, ambiguity, and information boundaries.
  • Prepared the initial business brief and annotated the client-held facts that agents could acquire through ORSim during formulation, solving, and refinement.
  • Supplied the hidden constraint verifiers, objective specifications, and original expert solutions.
  • Independently reviewed the consistency among original project requirements, agent-facing materials, hidden evaluators, and reference solutions.

Importantly, the construction process did not recast the original requirements into complete mathematical formulations or add new constraints absent from the original projects, prematurely disclose client-held facts, or introduce structural insights from subsequent expert analysis into the agent-facing materials (Appendix G.2).

These procedures were designed to preserve the information gaps and clarification requirements of the original engagements rather than simplifying them into fully specified optimization problems. The benchmark also retains sequential decision requirements where relevant. For example, Task 24 requires hourly decisions, with future data windows disclosed progressively as execution advances rather than being available at the outset. We further examine whether these design goals are reflected in actual agent interactions.

Information-recovery analysis demonstrates progressive information acquisition.

We supplemented the evaluation with an information-recovery analysis that measures how much of the required client information agents successfully acquire through interaction. This analysis covers five models (DeepSeek-V4.1-Flash, GLM-5.3-Flash, GPT-5.6 Sol, Grok 4.6, and Claude Opus 5) across all 31 instances, with five independent runs per model–instance combination. It uses trajectories from the main continuous-interaction setting, in which ORSim remains accessible throughout the OR workflow. Information recovery is calculated as the proportion of required non-brief facts that agents have completely and correctly obtained from the replies received up to a given interaction round. Critical-fact recovery applies the same calculation to a predefined subset of critical facts.

The results are summarized below.

Interaction pointOverall fact recovery (%)Critical-fact recovery (%)
After 1 round8.1212.63
After 5 rounds38.4553.26
After 10 rounds66.2878.42
At the end of each run81.0087.60

[1] Ho, Anson, et al. "A rosetta stone for ai benchmarks." arXiv preprint arXiv:2512.00193 (2025).

AuthorsOpenReview ↗

Response to Reviewer Aui7 (Part 2/10)

These results show that agents must progressively identify information gaps and acquire client facts through targeted questions. Despite continuous client access, average critical-fact recovery reaches only 87.60% at completion. This suggests that information availability alone does not guarantee successful acquisition: agents must identify information gaps and ask effective clarification questions. The benchmark evaluates both information use and the progressive clarification of business requirements. We have added this analysis to the revised manuscript in Appendix B.2.

Independent expert assessment of ORSim's interaction fidelity.

To independently assess whether ORSim reproduces realistic client-interaction characteristics, we recruited five additional OR practitioners, each with at least two years of industry experience and none of whom participated in benchmark construction. Each expert personally interacted with ORSim acting as the client across all 31 instances, producing 155 task-interaction records.

After completing each interaction, experts rated four dimensions on a 1–5 scale, yielding 620 dimension-level ratings. The ratings measure how closely ORSim's behavior resembles that of real clients: 1 indicates a poor match, while 5 indicates a close match. Importantly, higher scores indicate greater fidelity to real client behavior, not greater ambiguity or information incompleteness.

Evaluation dimensionWhat was assessedRating (mean ± SD) ↑Ratings of 4 or 5 (%) ↑
Requirement ambiguityWhether requirements exhibit realistic informal, ambiguous, or underspecified expressions requiring clarification4.68 ± 0.5496.13%
Information incompletenessWhether information gaps reflect what real clients may omit, assume to be known, or leave unspecified4.57 ± 0.5994.84%
Progressive disclosureWhether targeted follow-up questions lead to appropriate additional information and requirement clarification4.53 ± 0.6194.19%
Communication difficultyWhether terminology differences, implicit assumptions, and repeated clarification needs resemble realistic project communication4.48 ± 0.6392.90%

The four dimensions received consistently high ratings, with means ranging from 4.48 to 4.68. Ratings of 4 or 5 accounted for 92.90%–96.13% of ratings per dimension. These assessments provide independent practitioner evidence that ORSim preserves important features of real client information exchange, rather than merely serving as a mechanism for retrieving predefined facts.

Together, the information-recovery analysis and independent expert assessment provide complementary evidence: the former examines how information is acquired during actual agent interactions, while the latter evaluates how closely the simulated client interaction resembles real business communication. We have added the new expert assessment to the Appendix C.5 in the revised manuscript.

Distinctive evaluation value beyond general model capabilities.

To examine whether OR-LICENSE provides distinctive evaluation information beyond general-purpose capability rankings, we use the Epoch Capabilities Index (ECI) [1] as an external reference. ECI is a composite general-capability measure constructed by integrating results from more than 50 benchmarks into a common capability scale. We matched seven models with available ECI scores and examined their relationship with mean performance across the 31 OR-LICENSE instances. ECI and DAQ exhibit a moderate positive rank correlation, while substantial differences remain among models with similar ECI scores. For the seven models, there are (72)=7×6/2=21\binom{7}{2}=7\times6/2=21 distinct unordered model pairs. Among these pairs, six exhibit opposite rankings between ECI and DAQ. The corresponding rank correlation is reported below.

ComparisonSpearman's ρ
ECI vs. DAQ0.643

[1] Ho, Anson, et al. "A rosetta stone for ai benchmarks." arXiv preprint arXiv:2512.00193 (2025).

AuthorsOpenReview ↗

Response to Reviewer Aui7 (Part 3/10)

For example, Grok 4.6 and Muse Spark 1.3 differ by only 0.06 in ECI, yet their DAQ scores are 0.824 and 0.696, respectively, with feasibility rates of 83.23% and 70.97%. We have added the comparison between ECI and OR-LICENSE performance to the Appendix F.2 of the revised manuscript.

These results indicate that general-purpose capability rankings do not fully capture differences in end-to-end OR performance. In particular, models with similar general-capability estimates can differ substantially in their ability to deliver feasible, expert-relative OR solutions. By evaluating the complete process from acquiring incomplete business information and formulating optimization problems to producing validated solutions, OR-LICENSE provides a distinctive evaluation perspective that is not fully reflected in general-purpose benchmark aggregates.

[1] Ho, Anson, et al. "A rosetta stone for ai benchmarks." arXiv preprint arXiv:2512.00193 (2025).

Comparison with different information interaction mechanism.

To further assess the benchmark’s distinctive evaluation value, we compared three information access settings: full information upfront, initial clarification only, and continuous interaction. These comparisons examine whether the information acquisition requirements in the curated tasks have a measurable effect on final delivery performance.

  • Full information upfront: All disclosable business information is provided at the start of the task. No client queries are allowed.
  • Initial clarification only: Agents may inspect the supplied materials and query ORSim during initial clarification. Client querying is permanently disabled immediately before the agent first writes or modifies code implementing an optimization model or solution algorithm, or first invokes a solver or optimization routine, whichever occurs first. Reading files and inspecting their schema or summary statistics do not trigger this cutoff.
  • Continuous interaction (main setting): Agents can interleave client queries with formulation, execution, and revision, asking further questions as new information needs arise.

All three settings cover the same 11 models and all 31 tasks, with five runs per model–task pair in each setting. The table reports feasibility (%) and DAQ. The final row gives the equally weighted mean across the 11 models.

ModelFull upfront: Feasibility (%)Full upfront: DAQInitial only: Feasibility (%)Initial only: DAQContinuous: Feasibility (%)Continuous: DAQ
Claude Opus 550.32 ± 6.2952.45 ± 6.2735.48 ± 9.1237.61 ± 9.1943.87 ± 38.9544.07 ± 39.08
DeepSeek-V4.1-Flash58.06 ± 9.4057.22 ± 9.9947.74 ± 11.5047.17 ± 10.6552.90 ± 30.6552.21 ± 30.12
Doubao-Seed-2.1-Pro56.13 ± 1.7757.88 ± 1.6132.26 ± 6.0331.39 ± 5.4162.58 ± 3.6862.55 ± 2.61
Gemini 3.8 Flash75.48 ± 7.4377.50 ± 8.1060.00 ± 2.8962.31 ± 3.5668.39 ± 5.7768.60 ± 3.96
GLM-5.3-Flash67.74 ± 6.4567.90 ± 6.9067.74 ± 5.1067.79 ± 4.6561.94 ± 5.7761.49 ± 6.74
GPT-5.6 Sol76.13 ± 4.8980.53 ± 4.4861.29 ± 9.6865.27 ± 10.2369.03 ± 13.9971.58 ± 14.75
Grok 4.692.26 ± 3.6891.36 ± 4.1581.94 ± 5.4081.41 ± 6.2883.23 ± 7.7082.40 ± 9.78
Hy375.48 ± 2.8972.23 ± 3.7440.00 ± 5.4039.14 ± 4.7543.23 ± 7.0741.41 ± 5.75
Hy4 preview76.13 ± 6.6977.63 ± 6.6246.45 ± 7.4345.96 ± 7.9751.61 ± 17.9651.01 ± 19.71
Muse Spark 1.378.71 ± 4.8979.47 ± 3.7556.77 ± 2.8956.69 ± 3.5470.97 ± 5.5969.56 ± 4.17
Qwen3.8-Max63.23 ± 8.1065.50 ± 8.6948.39 ± 4.5650.28 ± 4.5256.13 ± 13.6156.48 ± 15.51
Equal-weight mean69.9770.8852.5553.1860.3560.12
Reviewer_ga74OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates LLM agents on 31 instances across 10 industrial scenarios derived from real consulting engagements. Agents acquire requirements through ORSim and produce executable solutions, evaluated using feasibility and objective quality relative to client-accepted expert deliveries. The benchmark addresses a meaningful gap between solving well-specified optimization problems and completing practical OR workflows.

Critical strengths
  1. Original business materials, incomplete requirements, and client-accepted reference solutions provide a valuable setting for evaluating requirement elicitation, modeling, and implementation together.
  2. Independent feasibility checks and objective recomputation reduce reliance on self-reported results. ORSim’s controlled fact disclosure and repeated-query reliability evaluation strengthen the interactive environment.
Critical weaknesses
  1. Experimental reporting needs reconciliation. Section 4 states that each model–task combination receives five runs. With 31 equally weighted instances and binary feasibility, mean feasibility should be an integer multiple of 1/155. Several Table 1 entries, including GPT-5.6 Sol’s 69.89%, are incompatible with this protocol after rounding. The reference to “available benchmark results” suggests incomplete coverage or different run counts, but these are not sufficiently documented. Additionally, the main text describes a matched harness ablation, whereas Appendix D.7 explicitly states that Table 7 should not be interpreted as a paired comparison. These inconsistencies limit confidence in the reported comparisons.

  2. The benefit of sustained client interaction is not directly established. RQ2 reports question timing and issue-resolution proportions, but lacks matched comparisons between full information upfront, initial clarification only, and unrestricted interaction. Later questions often target known problems, while early questions gather requirements, making their resolution rates difficult to compare directly. ORSim’s reliability evaluation is useful, but covers only five instances and does not quantify the downstream impact of simulator errors.

Questions for authors

1.What are the exact run counts, task coverage, and missing-run policies underlying Table 1? 2.Can the harness comparison be recomputed using genuinely matched experimental conditions? 3.How does downstream performance change when information is supplied upfront or interaction is restricted to initial clarification?

9 comments
AuthorsOpenReview ↗

Response to Reviewer ga74 (Part 1/6)

We sincerely appreciate your recognition of the significance of our work, particularly the value of evaluating LLM agents on real-world OR workflows and the meaningful gap our benchmark addresses beyond well-specified optimization problems. We are also grateful for your positive assessment of our evaluation design. We address your concerns regarding experimental reporting, comparisons across different information-interaction settings, and further analysis of ORSim's reliability through new experiments and revisions below.

W1&Q1: Experimental reporting

We apologize for the confusion caused by the lack of clarity in our experimental reporting. We clarify the exact run counts, task coverage, and missing-run policy underlying the original Table 1.

Run counts and policy.

Five runs were conducted for each model–task combination, yielding 155 total runs per model across 31 instances. However, some runs encountered API provider interruptions. Treating these failures as agent errors and counting them as infeasible outcomes would be inconsistent with our evaluation objective. We therefore excluded these runs from the results in the original Table 1. The original results were computed by averaging over valid runs within each instance, followed by an equally weighted average across the 31 instances.

Therefore, when some instances have fewer than five valid runs due to infrastructure errors, the resulting mean is not necessarily an integer multiple of 1/1551/155.

The exact run counts and aggregation results underlying the original Table 1 are provided below.

ModelValid-run distribution (instances × runs)Included / Total RunsFeasibility (%)QualityDAQ
Claude Opus 531×5155/15543.87 ± 38.95100.38 ± 1.4744.07 ± 39.08
DeepSeek-V4.1-Flash29×5 + 2×4153/15552.42 ± 31.0998.98 ± 2.1251.67 ± 30.45
Doubao-Seed-2.1-Pro29×5 + 1×4 + 1×3152/15562.37 ± 3.72100.09 ± 2.1462.37 ± 2.71
Gemini 3.8 Flash31×4124/15567.74 ± 6.45100.89 ± 4.8068.18 ± 4.44
GLM-5.3-Flash29×5 + 2×3151/15562.37 ± 4.9399.16 ± 4.7361.90 ± 6.64
GPT-5.6 Sol28×5 + 1×3 + 2×2147/15569.89 ± 14.55104.96 ± 4.4673.62 ± 16.95
Grok 4.630×5 + 1×3153/15583.87 ± 8.5399.43 ± 4.6183.58 ± 11.54
Hy330×5 + 1×3153/15544.09 ± 8.1296.18 ± 2.2542.28 ± 6.85
Hy4 preview31×5155/15551.61 ± 17.9698.01 ± 3.6951.01 ± 19.71
Muse Spark 1.331×5155/15570.97 ± 5.5998.12 ± 1.9569.56 ± 4.17
Qwen3.8-Max30×5 + 1×3153/15556.99 ± 13.04100.14 ± 3.3557.36 ± 14.98
Results with complete five valid runs

We further conducted runs to replace those affected by API provider failures and obtained five valid runs for every model–task combination. This gives a complete evaluation of 31 instances × 5 runs for each model. The resulting metrics are shown below.

ModelFeasibility (%)Quality (%)DAQ (%)
Claude Opus 543.87 ± 38.95100.38 ± 1.4744.07 ± 39.08
DeepSeek-V4.1-Flash52.90 ± 30.6598.89 ± 1.7852.21 ± 30.12
Doubao-Seed-2.1-Pro62.58 ± 3.68100.03 ± 2.1562.55 ± 2.61
Gemini 3.8 Flash68.39 ± 5.77100.53 ± 4.2468.60 ± 3.96
GLM-5.3-Flash61.94 ± 5.7799.25 ± 4.6861.49 ± 6.74
GPT-5.6 Sol69.03 ± 13.99103.62 ± 2.2971.58 ± 14.75
Grok 4.683.23 ± 7.7098.88 ± 4.1582.40 ± 9.78
Hy343.23 ± 7.0796.06 ± 2.4041.41 ± 5.75
Hy4 preview51.61 ± 17.9698.01 ± 3.6951.01 ± 19.71
Muse Spark 1.370.97 ± 5.5998.12 ± 1.9569.56 ± 4.17
Qwen3.8-Max56.13 ± 13.61100.10 ± 3.3556.48 ± 15.51

Our main conclusions from the result remain unchanged. Thank you for pointing this out. We have replaced Table 1 in the revised manuscript with the results based on five valid runs for every model–task combination, ensuring consistent task coverage and run counts across all models.

AuthorsOpenReview ↗

Response to Reviewer ga74 (Part 2/6)

W1&Q2: Matched harness ablation

We thank the reviewer for pointing out the inconsistency in our harness comparison. The original comparison was described as non-paired because occasional API provider interruptions resulted in different numbers of valid runs across model–task combinations. To address this issue, we reran the affected combinations until each model–task pair had five valid runs under both harnesses. In the updated comparison, we kept the model endpoints and versions, reasoning configurations, task inputs, ORSim settings, execution budgets, and verifier/evaluator versions consistent, with the agent harness as the intended experimental difference. We applied the same evaluation metrics and aggregation procedures, reporting mean and sample standard deviation across five benchmark-level repetitions.

Furthermore, to provide a more comprehensive assessment of the harness ablation, we extended the comparison to three additional models, resulting in six models evaluated under both Codex and Claude Code. This allows us to assess whether the main evaluation findings remain consistent across different agent harnesses.

The results for Codex are shown below.

ModelFeasibility (%)Quality (%)DAQ (%)
DeepSeek-V4.1-Flash69.03 ± 4.8999.15 ± 3.2768.34 ± 3.41
Doubao-Seed-2.1-Pro60.65 ± 4.78100.34 ± 1.2160.83 ± 4.63
GLM-5.3-Flash21.94 ± 12.3794.98 ± 13.3120.40 ± 11.31
GPT-5.6 Sol78.06 ± 10.55102.29 ± 1.7779.89 ± 11.18
Grok 4.688.39 ± 4.8997.92 ± 0.8386.52 ± 4.08
Muse Spark 1.373.55 ± 4.2194.44 ± 1.1969.43 ± 3.36

The two harnesses exhibit substantial agreement in DAQ-based model rankings. Across the six shared models, the rankings are positively correlated (Spearman's ρ=0.829\rho=0.829; Kendall's τ=0.733\tau=0.733), with 13 of the (62)=15\binom{6}{2}=15 unordered model pairs preserving the same ordering.

At the individual model level, Grok, GPT, and DeepSeek have higher mean DAQ under Codex, whereas Muse, Doubao, and GLM have higher mean DAQ under Claude Code. These differences suggest that harness choice can affect individual model performance, but the overall model ranking remains largely consistent.

More importantly, both harnesses support the central finding of OR-License: end-to-end feasibility remains the primary limitation, while conditional objective quality is relatively close to the expert reference. All six models remain below the expert reference in DAQ under both harnesses.

Rather than determining which harness performs better, this comparison examines whether the main evaluation findings persist across different agent execution environments. The expanded comparison, with consistent task coverage and run counts, strengthens confidence in the robustness of these findings while addressing the reporting inconsistency in the original ablation.

AuthorsOpenReview ↗

Response to Reviewer ga74 (Part 3/6)

W2&Q3: Comparisons across information access settings

Thank you for this suggestion. We agree that comparing full information upfront, initial clarification only, and continuous interaction provides a more direct answer to RQ2.

Continuous interaction is designed to reflect how client information is acquired in real OR projects: agents may identify new information gaps during formulation, solving, and validation, and seek further clarification from the client. Its value lies in incorporating this process into the evaluation. RQ2 examines how these information access conditions affect final solution feasibility and objective quality, including how the effects vary across models.

We therefore added two settings, full information upfront and initial clarification only, and compared them with continuous interaction, our main setting:

  • Full information upfront: All disclosable business information is provided at the start of the task. No client queries are allowed.
  • Initial clarification only: Agents can inspect the supplied materials and query ORSim during initial clarification. Client querying is permanently disabled immediately before the agent first writes or modifies code implementing an optimization model or solution algorithm, or first invokes a solver or optimization routine, whichever occurs first. Reading files and inspecting their schema or summary statistics do not trigger this cutoff.
  • Continuous interaction (main setting): Agents can interleave client queries with formulation, execution, and revision, asking further questions as new information needs arise.

All three settings cover the same 11 models and all 31 tasks, with five runs per model–task pair in each setting. The table reports feasibility (%) and DAQ. The final row gives the equally weighted mean across the 11 models.

ModelFull upfront: Feasibility (%)Full upfront: DAQInitial only: Feasibility (%)Initial only: DAQContinuous: Feasibility (%)Continuous: DAQ
Claude Opus 550.32 ± 6.2952.45 ± 6.2735.48 ± 9.1237.61 ± 9.1943.87 ± 38.9544.07 ± 39.08
DeepSeek-V4.1-Flash58.06 ± 9.4057.22 ± 9.9947.74 ± 11.5047.17 ± 10.6552.90 ± 30.6552.21 ± 30.12
Doubao-Seed-2.1-Pro56.13 ± 1.7757.88 ± 1.6132.26 ± 6.0331.39 ± 5.4162.58 ± 3.6862.55 ± 2.61
Gemini 3.8 Flash75.48 ± 7.4377.50 ± 8.1060.00 ± 2.8962.31 ± 3.5668.39 ± 5.7768.60 ± 3.96
GLM-5.3-Flash67.74 ± 6.4567.90 ± 6.9067.74 ± 5.1067.79 ± 4.6561.94 ± 5.7761.49 ± 6.74
GPT-5.6 Sol76.13 ± 4.8980.53 ± 4.4861.29 ± 9.6865.27 ± 10.2369.03 ± 13.9971.58 ± 14.75
Grok 4.692.26 ± 3.6891.36 ± 4.1581.94 ± 5.4081.41 ± 6.2883.23 ± 7.7082.40 ± 9.78
Hy375.48 ± 2.8972.23 ± 3.7440.00 ± 5.4039.14 ± 4.7543.23 ± 7.0741.41 ± 5.75
Hy4 preview76.13 ± 6.6977.63 ± 6.6246.45 ± 7.4345.96 ± 7.9751.61 ± 17.9651.01 ± 19.71
Muse Spark 1.378.71 ± 4.8979.47 ± 3.7556.77 ± 2.8956.69 ± 3.5470.97 ± 5.5969.56 ± 4.17
Qwen3.8-Max63.23 ± 8.1065.50 ± 8.6948.39 ± 4.5650.28 ± 4.5256.13 ± 13.6156.48 ± 15.51
Equal-weight mean69.9770.8852.5553.1860.3560.12

The comparison between continuous interaction and initial clarification only directly addresses RQ2’s question about the effect of continued information access on final performance. Averaged equally across the 11 models, continuous interaction increases feasibility from 52.55% to 60.35%, a gain of 7.80 percentage points, and DAQ from 53.18 to 60.12, a gain of 6.94 points. Ten of the eleven models achieve higher mean feasibility and DAQ. These results show that continued access to client information improves final delivery performance for the vast majority of evaluated models and yields a positive average gain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank all three reviewers for recognizing the significance of OR-License and the value of its design. Reviewer ga74 highlighted “a meaningful gap between solving well-specified optimization problems and completing practical OR workflows” and positively assessed our evaluation design. Reviewer Aui7 recognized its “strong real-world relevance” and “significant practical value,” particularly its comprehensive end-to-end framework integrating real-world data curation, client interaction, solution verification, and expert-reference evaluation. Reviewer 9Aim emphasized the novelty and importance of our dataset derived from real industrial OR projects, as well as the design of ORSim.

OR-License represents an important shift in the LLM-for-OR evaluation paradigm. To the best of our knowledge, it is the first benchmark to assess whether LLM agents can independently define problems from ambiguous, incomplete real-world business requirements and undertake the end-to-end responsibilities of OR consultants, rather than merely solve predefined optimization problems. This shift is directly relevant to industrial use: an agent that only solves problems fully specified by experts can undertake only part of professional OR work. OR-License measures this broader capability and reveals how far current models remain from professional-level OR delivery. To support this evaluation, we built 31 project-level tasks from real paid OR consulting projects, covering 10 industries and worth a combined $3.68 million. Each task has its own dedicated evaluation environment with hidden feasibility verifiers and objective-quality evaluators.

We summarize the eight revisions and clarifications during the rebuttal below. All changes are highlighted in the revised PDF.

  1. Information access. We compared full upfront, initial-only, and continuous access across 11 models and 31 tasks (five runs per pair). Continuous access raised mean feasibility 52.55%→60.35% and DAQ 53.18→60.12 over initial-only (10/11 improved); full upfront reached 69.97%/70.88. This tests RQ2 through final delivery, beyond question timing (Sec. 4.2; Table 2; App. D.8).

  2. Real-world fidelity. In OR-License, agents recovered 8.12% of required facts missing from the brief after one round of questions and 81.00% by the end of the run. Recovery of critical facts rose from 12.63% to 87.60%. Five practitioners evaluated ORSim on 31 tasks (155 interactions, 620 ratings), scoring fidelity 4.48–4.68/5. These assess interaction realism, information access, and agent recovery (Apps. B.2, C.5, G.2).

  3. ORSim reliability. Testing now covers all 31 tasks (1,244 queries; 24,880 replies); an expert-audited judge agreed with experts on 99.23% (258/260; α=0.981). Correcting the first confirmed error in 133 affected runs changed feasibility 59.40%→60.15% and DAQ 60.67→61.39, but reduced ask calls 18.26→10.90. Errors affected interaction cost more than final scores in this test (Apps. C.2–C.3).

  4. Experimental reporting and reproducibility. We completed the evaluation coverage with five valid runs for every model–task pair and updated Table 1 accordingly. We also clarified the valid-run counting and instance-equal aggregation procedures (Sec. 4.1; App. D.3).

  5. Disclosure cap. We tested caps of 1, 2, 3, or 4 facts per reply and a setting with no numerical cap. Raising the cap from 1 to 3 increased fact recovery from 68.4% to 81.0%, reduced critical-fact omissions from 25.6% to 12.4%, and cut mean queries per run from 29.7 to 14.3. Removing the three-fact cap added only 0.8 percentage points of recovery while raising irrelevant disclosure from 3.0% to 14.5%, supporting the three-fact default (App. C.4).

  6. DAQ and weighting. DAQ gates expert-relative quality on strict feasibility, giving infeasible deliveries zero credit; it is not a standalone deployment criterion. Strict Feasibility and Conditional Quality remain separate, while the DAQα profile exposes their trade-off. Equal instance weights avoid test-case count bias without assuming equal task difficulty or value. All models trail the expert in DAQ. Ten-industry results reveal heterogeneity (App. D.9).

  7. Harness robustness. We compared six models across Codex and Claude Code, aligning versions, reasoning settings, budgets, and evaluation. DAQ rankings largely agree (Spearman ρ=0.829; Kendall τ=0.733), despite model-level differences. Both identify feasibility, not conditional quality, as the main bottleneck (App. D.7).

  8. Beyond general capabilities. DAQ and Epoch Capabilities Index have only moderate rank correlation (ρ=0.643). The model rankings differed on 6 of 21 model pairs. General rankings thus miss OR delivery differences (App. F.2).

These additional experiments and analyses provide further empirical validation of OR-License’s real-world fidelity, evaluation reliability, and distinctive evaluation value. We are grateful for the reviewers' insightful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.