- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces OR-License, a benchmark for evaluating whether LLM agents can perform end-to-end operations research (OR) tasks using original client materials rather than fully specified optimization problems. The benchmark contains 31 instances across 10 industrial scenarios drawn from paid OR projects, with client-accepted expert solutions serving as references. It also introduces ORSim, a client simulator that allows agents to request additional business information throughout the optimization process without receiving OR expertise. Agent submissions are evaluated using hidden feasibility checks and expert-relative objective quality, including a feasibility-gated metric called Deployment-Adjusted Quality (DAQ). The authors evaluate 11 LLMs using coding-agent frameworks and examine their information-seeking behavior, research processes, and failure patterns. The results show that some models outperform the expert references in objective quality on feasible instances, but none matches the expert reference in DAQ. The best-performing system achieves 83.87% strict feasibility and 83.58% DAQ. The paper also reports a separate reliability evaluation of ORSim and analyzes common difficulties in translating business requirements into feasible optimization solutions.
Critical strengths
The main strength of this paper is the construction of a benchmark from real industrial OR projects. OR-License contains 31 instances across 10 industrial scenarios, with original client requirements, input data, and expert solutions accepted by clients before benchmark construction. Unlike benchmarks based on predefined optimization problems, it evaluates whether agents can interpret business requirements and produce executable solutions. This addresses an important gap in existing OR benchmarks and provides a valuable resource for studying LLM agents in professional settings.
Another strength is the design of ORSim, which allows agents to acquire business information throughout the optimization process while restricting access to OR expertise. The separation of fact selection, deterministic authorization, and response generation makes the interaction more controlled and auditable. The authors also evaluate the simulator's reliability using 204 expert-annotated queries with repeated trials. Together with hidden feasibility checks and expert-relative objective evaluation, this provides a reasonably well-designed framework for assessing end-to-end OR performance.
Critical weaknesses
My concern is that the experiments do not directly answer RQ2 as stated. The paper aims to examine how allowing agents to acquire client information throughout optimization affects solution feasibility and objective quality. However, Section 4.2 and Figure 5 mainly report when agents ask questions and the proportion of questions associated with confirmed issue resolution. There is no controlled comparison between agents that can ask questions only during initial clarification and agents that can continue asking throughout the optimization process. The reliability experiments in Appendix C provide useful evidence about ORSim's response quality, but they do not establish whether continued access to client information improves final optimization performance. The current results support the observation that information acquisition extends beyond initial clarification, but they cannot establish its effect on feasibility or DAQ.
Questions for authors
The paper identifies information acquisition throughout optimization as a central contribution, and RQ2 aims to examine its effect on feasibility and objective quality. However, Section 4.2 mainly reports the timing of client questions and their confirmed issue-resolution rates. Could the authors clarify whether RQ2 is intended to characterize information-seeking behavior or to establish the benefit of continued access to ORSim? If the latter, is there evidence comparing agents that can ask questions only during initial clarification with agents that can continue asking throughout optimization? This distinction would affect how strongly the experimental results support the claimed contribution.
Appendix G.2 describes expert reviews of the consistency between the original requirements, agent-facing materials, and hidden evaluators. Could the authors clarify how they verified that all business requirements enforced by the hidden evaluators could be recovered from the initial materials, input data, or authorized ORSim responses? In particular, was this information coverage systematically checked for all 31 instances? This is important for determining whether failures primarily reflect limitations in the agents' ability to understand and formulate OR problems, rather than information that was not accessible during evaluation.
5 comments
Response to Reviewer 9Aim (Part 2/2)
Model-level results also reveal variation: GLM-5.3-Flash shows lower mean feasibility and DAQ under continuous interaction, and the relative rankings of models also change. When ranked by mean DAQ, eight of the eleven models change position, with Doubao rising from 11th to 5th and Muse from 5th to 3rd. Together, these results show that the gains from continued access to client information vary across models, and that information access conditions affect assessments of their relative performance.
We have incorporated this comparison into Section 4.2 of the revised paper, using changes in final feasibility and DAQ as the primary evidence for answering RQ2. Figure 5 provides a behavioral analysis of when agents seek information and the associated evidence of issue resolution, while Appendix C continues to report the reliability of ORSim’s responses. Together, these analyses present evidence on final performance, the interaction process, and simulator reliability.
Q2: Systematic Information-Coverage Validation
We thank the reviewer for raising this important question. During benchmark construction, we systematically checked information coverage for all 31 instances. Specifically, for each instance, the responsible domain expert checked the business requirements enforced by the hidden evaluator individually and verified that the information needed for each requirement could be obtained through the same channels available to evaluated agents: the initial business brief, input data, and direct interaction with ORSim. For requirements requiring client clarification, the experts also queried ORSim themselves to confirm that the relevant facts could be obtained through authorized responses.
Furthermore, we recruited five additional OR practitioners with at least two years of industry experience who were not involved in benchmark construction to independently validate information accessibility. Each expert personally interacted with ORSim acting as the client and assessed whether the business requirements enforced by the hidden evaluators could be recovered from the initial materials, input data, and authorized ORSim responses.
| Evaluation dimension | Rating (mean ± SD) ↑ | Pairwise Exact Agreement ↑ | 95% Bootstrap CI | Verified information accessibility ↑ |
|---|---|---|---|---|
| Accessibility of business requirements | 4.99 ± 0.11 | 98.06% | [94.19%, 100.00%] | 100% |
The Verified information accessibility measures the proportion of predefined, authorized information items that experts successfully acquired through direct interaction with ORSim, aggregated across expert–instance combinations. A value of 100% requires both complete acquisition of the audited information items and expert confirmation that the information needed for every hidden evaluation requirement is accessible. We have added the systematic information-coverage validation across all 31 instances to Appendix G.2 and the independent expert assessment of information accessibility to Appendix C.5 of the revised manuscript.
Together, these results show that information accessibility was systematically verified during task construction for all 31 instances and independently re-examined by five practitioners who had not participated in benchmark construction. Both checks provide evidence that the business requirements enforced by the hidden evaluators can be recovered through the information channels available to agents, including requirements that require active client questioning. However, information being accessible does not guarantee that agents will successfully acquire and use it. Agents must still identify information gaps, ask effective questions, and translate the acquired business requirements into correct optimization formulations and executable solutions. These are central capabilities that OR-License aims to evaluate.
Dear Reviewer,
We hope you are well. We would like to express our thanks once again for your helpful comments. As the discussion period draws to a close, we wanted to follow up and see whether our responses have satisfactorily addressed the points you raised. Please feel free to let us know if any aspect would benefit from further clarification. We appreciate your insights and would be glad to address any additional feedback.
Thank you again for your thoughtful review and the time you have dedicated to our paper.
Response to Reviewer 9Aim (Part 1/2)
We sincerely thank the reviewer for recognizing the novelty and importance of OR-LICENSE, particularly its dataset contribution based on real industrial OR projects, and the design of ORSim. We appreciate your insightful comments on the distinction between characterizing agents' information-seeking behavior and establishing the benefits of sustained client interaction. We address your concerns through two complementary evaluations: (1) a comparison of three client-information interaction settings to examine their effects on final solution performance, and (2) a systematic information-coverage validation across all 31 instances to verify that the requirements enforced by hidden evaluators are recoverable from the information available to agents. We detail these additional evaluations and their findings below.
W1&Q1: Comparisons across different information-interaction settings
Thank you for this suggestion. RQ2 aims to evaluate how continued access to client information affects final optimization performance. We agree that a direct comparison between initial clarification only and continuous interaction is needed to assess the effect of retaining access to ORSim after the initial phase.
To this end, we compared the following two settings across the same 11 models and all 31 tasks, with five runs per model–task pair in each setting:
We evaluated final submissions using strict feasibility and DAQ. DAQ incorporates objective quality relative to the expert reference and assigns zero credit to infeasible submissions. The results are shown below, with the final row reporting the equally weighted mean across the 11 models.
Continuous interaction increases average feasibility from 52.55% to 60.35%, a gain of 7.80 percentage points, and average DAQ from 53.18 to 60.12, a gain of 6.94 points. Ten of the eleven models achieve higher mean feasibility and DAQ. These results show that, in this evaluation, retaining access to client information after initial clarification improves final delivery performance for the vast majority of evaluated models and yields a positive average gain.