acceptodds
Under review as a conference paper at ICLR 2027

OR-Space: A Workspace-Based Benchmark for Industrial Optimization Agents

Abstract

Large language model (LLM) agents are increasingly used for operations research (OR) modeling, but existing OR benchmarks mostly test model construction from a self-contained problem statement. Industrial optimization also requires connecting requirements to data and code, revising models, and explaining solver results. We introduce OR-Space, a benchmark of executable workspaces containing business documents, structured data, code, solver outputs, and task-specific evaluators. OR-Space evaluates three tasks separately: BUILD, where agents create solver-ready models from heterogeneous artifacts; REVISE, where they update models under changed requirements while preserving valid prior logic; and EXPLAIN, where they ground answers about solutions and constraints in workspace evidence. OR-Space evaluates 16 language models on 100 base problems instantiated as 300 task instances, revealing substantial failures in schema mapping, constraint grounding, legacy-code use, and solver-backed explanation. Controlled studies show that workspace interface, revision context, and solver backend affect performance.

Primary area:
optimization
Keywords:
large language models, optimization modeling, benchmarks, workspace reasoning, model revision, grounded explanation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_kXXAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OR-Space, a benchmark for evaluating LLM agents on industrial optimization tasks. Instead of presenting each problem as a self-contained description, OR-Space organizes the required information into executable workspaces containing business documents, structured data, code, and solver records.

The benchmark consists of three independently evaluated tasks: BUILD, REVISE, and EXPLAIN. It contains 300 task instances derived from 100 IndustryOR problems. The authors evaluate 16 language models and two agent harnesses, and study the effects of workspace organization, revision context, and solver backend.

Critical strengths
  1. The benchmark addresses a relevant gap in existing OR evaluations. Building an optimization model from business requirements and structured data, updating an existing implementation, and explaining solver results are different capabilities. Evaluating them separately is useful, particularly since many existing benchmarks focus primarily on initial model construction.

  2. The experimental analysis is reasonably comprehensive. The filesystem-versus-flat comparison is particularly interesting, showing that organizing the same information into separate files can make model construction more difficult. The comparisons across revision contexts and solver backends are also useful. The authors have made a substantial effort to validate the benchmark, including reference-model verification, failure analysis, expert calibration of the EXPLAIN evaluator, and practitioner assessment of task representativeness.

Critical weaknesses
  1. The BUILD and REVISE evaluation metric does not fully capture formulation correctness.

According to Section 4.1, a submission is considered successful when the generated program executes with the required solver and its objective value agrees with the reference optimum within a specified tolerance.

However, matching the optimal objective value does not necessarily mean that the generated mathematical model is correct. An agent may omit a constraint that is inactive at the reference optimum, or implement an incorrect feasible region, while still obtaining the expected objective value. This is particularly relevant for REVISE, where preserving valid modeling logic is explicitly part of the task.

The authors carefully validate the reference implementations during dataset construction. However, this does not establish the correctness of agent-generated models that pass the objective-based evaluation.

I would like to see some analysis of this potential false-positive issue, perhaps through manual inspection of passing submissions or testing generated models under modified parameter values. This would help establish how reliably Pass@1 reflects actual modeling capability.

  1. The EXPLAIN evaluator is reasonably designed, but its validation does not yet establish the reliability of model rankings.

I appreciate the expert calibration study in Appendix F. The comparison with human experts is useful and provides more evidence than an unvalidated LLM-as-a-judge setup.

However, the study covers 90 responses from only three models. The judge achieves a total-score MAE of 6.9 points and identifies 18 of the 24 responses containing substantive mathematical errors. These results are informative, but they do not establish whether the relative rankings remain stable across all 16 evaluated models.

I agree with the authors' own observation in Appendix F.4 that the calibration results are insufficient to validate the full ranking. Further analysis would strengthen the interpretation of the EXPLAIN results, especially for models with similar scores.

Questions for authors
  1. Have the authors examined whether any BUILD or REVISE submissions that pass the objective-based metric contain incorrect mathematical formulations? A manual audit of a sample of passing submissions, or evaluation under perturbed input parameters, would be particularly helpful.

  2. How robust are the EXPLAIN model rankings to the choice of evaluator? Would a different judge or additional expert evaluations produce similar rankings, especially among models with relatively close scores?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer kXXA (1/2): Formulation Correctness & Manual Audit [W1, Q1]

We thank Reviewer kXXA for the thoughtful and constructive feedback. In response to your two questions, we:

  1. Asked two OR experts to manually audit every passing BUILD and REVISE submission of Gemini 3.1 Pro, alongside an automated formulation check (Appendix A.7).

  2. Re-scored all 16 models on EXPLAIN with alternative judge models (Appendix C and Appendix F.6).


[W1, Q1] Formulation Correctness and Manual Audit of Passing Submissions

Evaluation boundary and the open problem of formulation equivalence.

We agree that matching the reference optimal objective on one input can occasionally pass a flawed formulation (e.g., omitting a nonbinding constraint or relaxing integer variables). We state this boundary explicitly in Section 4.1 and followed your suggestion to inspect passing submissions directly (Appendix A.7).

At the same time, automatically deciding whether two general LP/MILP formulations are mathematically equivalent remains an open problem, as one problem admits multiple equivalent formulations (e.g., variable substitutions, auxiliary variables, Big-MM vs. indicator constraints, or different constraint splits).

Much as math and coding benchmarks (GSM8K/MATH, SWE-bench) grade final answers or test outcomes rather than non-unique intermediate steps, solver execution with optimal-objective agreement remains the standard primary metric across OR benchmarks (ComplexOR, OptiMUS, OptiBench, IndustryOR, MIPLIB-NL), while recent verifiers (EquivaMap, ORGEval, SAC-Opt, Opt-Verifier) still lack general equivalence guarantees across multi-file solver code.

1. Complete manual audit by two OR experts (Appendix A.7, Table 14).

Two OR PhD researchers manually audited every Gemini 3.1 Pro submission that passed our objective metric across the 100-instance benchmark—covering all 72 BUILD passes and all 87 REVISE passes (159 programs; 318 independent reviews).

A program was flagged as a false positive if it omitted a required constraint, changed variable types or bounds, altered the feasible region, dropped an input parameter binding, or broke valid prior logic in REVISE (equivalent reformulations and helper variables were not penalized):

StageObjective-pass submissionsExpert A false positivesExpert B false positivesMean false-positive rate
BUILD722/72 (2.8%)3/72 (4.2%)3.5%
REVISE874/87 (4.6%)4/87 (4.6%)4.6%
Combined1596/159 (3.8%)7/159 (4.4%)4.1%

Across all 318 checks, 13 false-positive flags occurred (4.1% mean: 3.5% on BUILD and 4.6% on REVISE). While objective-only false positives do occur, their rate on Gemini 3.1 Pro is low (~4.1%); over 95% of passing submissions implement valid formulations (Appendix A.7).

2. Automated formulation check (SAC-Opt and Opt-Verifier, Appendix A.7, Tables 12–13).

We also adapted two ICML 2026 methods using gpt-5.6-sol (τa=0.7\tau_a = 0.7) on 20 frozen instances per model (40 outputs across Gemini 3.1 Pro and Claude Opus 4.6):

  • SAC-Opt (Zhang et al., 2026): checks whether code covers the stated variables, objectives, constraints, and parameters.

  • Opt-Verifier (Liu et al., 2026): evaluates model structure separately from solution validity.

Candidate modelNNObjective / executionSAC-Opt anchor meanSAC-Opt anchor passOpt-Verifier structureOpt-Verifier solutionOpt-Verifier joint
Gemini 3.1 Pro2015/20 (75.0%)0.857217/20 (85.0%)7/20 (35.0%)9/20 (45.0%)4/20 (20.0%)
Claude Opus 4.62013/20 (65.0%)0.811915/20 (75.0%)6/20 (30.0%)9/20 (45.0%)3/20 (15.0%)
All audited cases4028/40 (70.0%)0.834632/40 (80.0%)13/40 (32.5%)18/40 (45.0%)7/40 (17.5%)

SAC-Opt confirms high element coverage (80.0%), whereas strict single-reference structure matching is 32.5% (11/40 solution-correct / model-inconsistent cases in Table 13).

Because general formulation equivalence is an open problem and our reference model is only one valid representation among many, single-reference verifiers penalize valid equivalent formulations (11/40 valid-solution rejections vs. 4.1% manual false positives)—making solver-grounded Pass@1 a more direct primary metric (Section 4.1, Appendix A.7).

AuthorsOpenReview ↗

Author Response to Reviewer kXXA (2/2): EXPLAIN Rubric & Multi-Judge Consistency [W2, Q2]

[W2, Q2] EXPLAIN Rubric Design and Ranking Consistency Across Judge Models

1. Rubric structure (Appendix C, Table 20).

Our EXPLAIN rubric combines programmatic checks with structured criteria:

  • Deterministic Exact Coverage (35 / 100 pts): scored directly by code—matching required numbers, variable names, and CSV columns after text normalization, without calling an LLM judge.

  • Checklist & anchored criteria (65 / 100 pts): per-question checklist items (Reasoning, 35 pts) and explicit 0–3 scoring rules (Grounding, 20 pts; Answer Quality, 10 pts).

  • Unsupported-claim penalties (0 / 5 / 12 / 20 pts): require quoting the specific unsupported statement (0 penalty if none is quoted).

2. Re-scoring all 16 models with alternative judges (Appendix F.6, Table 23).

Keeping the rubric, prompts, and temperature fixed, we re-scored all 100 EXPLAIN instances across all 16 models using two non-evaluated models (gpt-5.4, gpt-5.6-sol), gemini-3.1-pro (testing self-preference), and a three-judge mean (gpt-5.1, gpt-5.4, gpt-5.6-sol):

Judge configurationSpearman ρ\rhoKendall τ\tauTop-5 overlapPairwise order (120 pairs)
gpt-5.40.990.935/5116/120 (96.7%)
gpt-5.6-sol0.980.904/5114/120 (95.0%)
gemini-3.1-pro0.980.884/5113/120 (94.2%)
Three-judge mean0.990.975/5118/120 (98.3%)

Rankings remain highly consistent across judges (Spearman ρ=0.98\rho = 0.98–0.990.99; pairwise order agreement 94.2%–98.3%). Paired bootstrap 95% CIs include zero for the 8 close pairs (< 1.5 pts apart).

Top-5 overlap is 4/5 or 5/5. On the 90-response expert set, gpt-5.4 (MAE = 6.5) and gpt-5.6-sol (7.1) closely track gpt-5.1 (6.9), and gemini-3.1-pro shifts its own mean by only +0.35 (rank 8 unchanged). Accordingly, we treat models within 2 points as tied (Section 4.1, Appendix F.6).


We hope these audits and supplementary results address your questions, and we welcome any further discussion.

Reviewer_iWHQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OR-Space, a workspace-based benchmark evaluating optimization agents on model construction, revision, and grounded explanation. It comprises 300 task instances derived from 100 IndustryOR problems and evaluates 16 language models under different interfaces and solver configurations.

Critical strengths

The main strength is the explicit evaluation of revision and explanation alongside initial modeling. The controlled experiments provide useful evidence of how workspace organization, available context, and solver interfaces affect agent performance.

Critical weaknesses

A major weakness is that the evidence is insufficient to support the claimed novelty of the workspace-based design: the filesystem comparison shows added difficulty, not a distinct capability. Also, the paper misses many existing literature, including comparisons with EOR, OptiChat, OptiBench, and FrontierOR, which leave the contribution beyond existing work unclear.

Questions for authors
  1. Could the authors clarify the novelty of the paper. Specifically, what the workspace design adds beyond file handling complexity?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer iWHQ (1/2): Workspace Design & Filesystem vs. Flat Study [W1, Q1]

We thank Reviewer iWHQ for the constructive feedback and for pointing out important related benchmarks in OR modeling and interactive OR agents. Below we clarify:

  1. Our multi-file workspace design, controlled Filesystem vs. Flat comparison, and evaluation on complex partner cases (HARD20).

  2. How OR-Space relates to EOR, OptiChat, OptiBench, FrontierOR, MIPLIB-NL, and ORAgentBench.


[W1, Q1] Workspace Design and Filesystem vs. Flat Comparison

1. Multi-file OR workspaces and reusable construction pipeline.

Much as code benchmarks moved from single-function tasks (HumanEval / MBPP) to multi-file repositories (SWE-bench), separating OR tasks into business documents (D\mathcal{D}), linked data tables (P\mathcal{P}), and existing solver code (S\mathcal{S}) tests whether agents can align table keys with mathematical variables, decouple model logic from data, and update existing code.

Beyond any fixed seed set, OR-Space provides a reusable four-step pipeline (business specification →\to linked CSV tables →\to solver-code workspace →\to hidden reference model, 10−310^{-3}-checked optimal value, and EXPLAIN checklist) so new industrial problems use the exact same BUILD–REVISE–EXPLAIN setup.

2. Controlled Filesystem vs. Flat comparison (Table 3 / Appendix A.2, Table 7).

Across six models on the exact same instances, data, solver, and scoring rules, we vary only inline prompt placement (Flat) vs. separate workspace files (Filesystem):

ModelBUILD FsBUILD FlatBUILD Δ\DeltaREVISE FsREVISE FlatREVISE Δ\DeltaEXPLAIN FsEXPLAIN FlatEXPLAIN Δ\Delta
gemini-3.1-pro727208794-769.1370.55-1.42
deepseek-v4-pro6770-38085-573.6273.76-0.14
gpt-5.55966-78589-485.0984.17+0.92
claude-sonnet-4.55362-98285-381.1179.68+1.43
qwen3-max4960-116875-772.9574.19-1.24
gpt-4o2544-193650-1456.1754.57+1.60
Average54.262.3-8.273.079.7-6.773.0172.82+0.2

Here, the accuracy gap is neither uniform across models nor uniform across tasks:

  • Larger drops on weaker models (0 to -19 pp): Uniform file-reading overhead would lower scores similarly across models. Instead, on BUILD, gemini-3.1-pro drops 0 pp (72% → 72%) and deepseek-v4-pro drops 3 pp, while weaker models drop progressively more (-7 →\to -9 →\to -11 →\to -19 pp). Consistently, 39% of BUILD errors (Section 4.6, Table 11) stem from cross-file column/key mapping vs. 11% from syntax errors.

  • Different impact across tasks: On the exact same workspaces, EXPLAIN (synthesizing cross-file evidence without writing multi-table code) is unchanged (+0.2 pts), whereas BUILD (-8.2 pp) and REVISE (-6.7 pp) decline across models.

AuthorsOpenReview ↗

Author Response to Reviewer iWHQ (2/2): Industrial HARD20 [W1, Q1] & Related Benchmarks [W2]

[W1, Q1] Larger Industrial Cases (HARD20, Appendix A.8, Table 16)

Applying the same four-step pipeline, we built HARD20 from 20 real, complex industrial cases provided by our industrial partners (desensitized problems across production changeovers, parallel processing, repair/assembly, hydrogen/chemical-energy planning, network security, and power-unit commitment; 5–16 linked CSVs, averaging 2,787 variables and 3,811 constraints, up to 9,373 and 15,193):

Solver backendModelBUILD Pass@1 (%)REVISE (code) Pass@1 (%)EXPLAIN Rubric (0–100)
Gurobi 13.0.1gpt-5.545.055.086.19
deepseek-v4-pro10.045.076.43
deepseek-v4-flash0.055.078.66
COPT 8.1.0gpt-5.530.040.087.47
deepseek-v4-pro20.010.074.37
deepseek-v4-flash0.00.078.02

Scaling from the 100-instance benchmark to 5–16 linked tables in HARD20 reveals two clear patterns:

  • Non-linear amplification with problem size: On the 100-instance Gurobi BUILD set, gpt-5.5, deepseek-v4-pro, and deepseek-v4-flash span 59.0% / 67.0% / 54.0% (13 pp spread). On HARD20, they diverge sharply to 45.0% / 10.0% / 0.0% (30.0% / 20.0% / 0.0% under COPT), with smaller models hitting 0% when reconstructing multi-period couplings from scratch.

  • BUILD vs. REVISE measure distinct capabilities: When given existing Gurobi code in REVISE (code), deepseek-v4-flash solves 55.0% (matching gpt-5.5), contrasting sharply with its 0.0% on BUILD. We will open-source HARD20 in a subsequent release.


[W2] Comparison with EOR, OptiChat, OptiBench, FrontierOR, and Related Benchmarks

Thank you for pointing out these four works. We updated Section 2, Section 3.3, Table 1, and Appendix D to include EOR, OptiChat, OptiBench, and FrontierOR alongside MIPLIB-NL and ORAgentBench:

  • Model interaction and explanation (EOR, OptiChat): EOR (Zhang et al., 2025) supports what-if edits and explanations via Decision Information, and OptiChat (Chen et al., 2025) studies sensitivity analysis, infeasibility diagnosis, and counterfactuals on pre-built models. OR-Space complements them by evaluating construction (BUILD), code updates (REVISE), and solver-backed explanation (EXPLAIN) together—both separately and sequentially (Appendix A.11).

  • Optimization modeling and algorithm design (OptiBench, FrontierOR): OptiBench (Yang et al., 2025) evaluates single-stage modeling from descriptions and tables, while FrontierOR (Kong et al., 2026) designs fast solving algorithms for large-scale optimization. OR-Space complements both by testing how agents build, update, and explain formulations across evolving requirements and linked files.

  • Industrial modeling and multi-file OR agents (MIPLIB-NL, ORAgentBench): MIPLIB-NL (Li et al., 2026) evaluates single-stage industrial modeling with separate data files, and ORAgentBench (Li et al., 2026) focuses on multi-file operational feasibility, solution quality, and dynamic replanning. OR-Space complements both with its four-step pipeline, paired BUILD–REVISE–EXPLAIN tasks, controlled revision contexts, and sequential evaluation.

Overall, OR-Space's contribution lies not in any single task in isolation, but in evaluating construction, revision, and explanation—both separately and sequentially—within one shared workspace benchmark. We also explicitly separate inherited seed problems from our workspace construction and evaluation setup in Section 3.3.


We hope these additional experiments and comparisons address your questions, and we would be happy to discuss further.

AuthorsOpenReview ↗

Your feedback would be appreciated

Dear Reviewer iWHQ,

We would like to sincerely thank you for your thoughtful and valuable feedback on our submission. We have carefully addressed each of your points and made revisions accordingly. We would greatly appreciate it if you could review the changes and let us know if there are any remaining concerns.

Once again, thank you for your time and support in reviewing our work. We look forward to hearing from you soon.

Reviewer_TwshOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an agent benchmark, OR-Space, focusing on evaluating the ability of LLMs to construct, modify, and interpret optimization models with real-world workflows. It constructs 300 tasks based on 100 problems and evaluates 16 language models, 2 agent frameworks, and solvers.

Critical strengths
  • The evaluation of OR agents is a meaningful research direction. The paper has a broad scope of experiments. This provides insights into the models and varying configurations.

  • The authors engaged OR experts to independently assess 90 of the generated responses, which provides an important perspective from human experts.

Critical weaknesses
  • The benchmark uses data from the existing IndustryOR dataset to reconstruct the original models and then generate operational files, parameters, modification requirements, and problem descriptions. The authors need to further clarify the novelty of the proposed method.

  • Evaluation based on the objective value may not be enough to guarantee correctness. Equation 2 shows that the authors consider tasks successful if the final objective value is sufficiently close to the reference value. However, this ​​does not imply that the underlying mathematical models are correct. Two models might have different constraints or decision-variable definitions, yet still yield the same optimal objective value for a given set of input data. So the agent may overlook new requirements but still pass the evaluation because the objective value remains unchanged.

Questions for authors
  1. For the new benchmark, does the added complexity (by modifying the original IndustryOR dataset) come from actual industrial business processes, or is it just a result of splitting files and imposing additional requirements?

  2. If a model has already been trained or tested on IndustryOR, how to minimize the risks of data contamination from reusing the same underlying optimization problems?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Twsh (1/3): Task Design, Pipeline, HARD20 & Filesystem vs. Flat [W1, Q1]

We thank Reviewer Twsh for the insightful and constructive feedback. Below we address:

  1. How our workspace tasks and four-step pipeline extend beyond IndustryOR to complex partner cases (HARD20).

  2. How our controlled Filesystem vs. Flat study separates task difficulty from multi-file layout.

  3. How automated and manual code audits verify all stated and newly added REVISE requirements.

  4. How we mitigate data-contamination risks from public seeds.


[W1, Q1] Task Design, Four-Step Construction Pipeline, and HARD20

1. Practitioner ratings and error breakdown (Appendix F.5, Table 22 & Section 4.6, Table 11).

Eight industrial OR practitioners blindly rated 30 workspaces at 4.1/5 on document realism (78% >= 4), 4.0/5 on revision realism (73% >= 4), and 4.3/5 on practical modeling skills (85% >= 4). Consistently, in BUILD failures (Table 11), 39% stem from cross-table key/column mapping and 50% (29% + 21%) from business-constraint modeling, vs. 11% from syntax errors.

2. Reusable four-step construction pipeline.

Beyond any fixed seed set, OR-Space provides a reusable pipeline that converts any seed optimization problem into a full workspace without changing task interfaces or scoring:

  • (i) Specification: authoring business documents and requirement-change scenarios;

  • (ii) Data generation: creating linked multi-table CSV/JSON files with cross-file keys;

  • (iii) Workspace packaging: organizing visible documents, tables, and solver templates;

  • (iv) Hidden verification: generating reference code, 10−310^{-3}-checked optimal values, and EXPLAIN checklists.

3. Real industrial cases from partners (HARD20, Appendix A.8, Table 16).

Using this unchanged pipeline, we built HARD20 from 20 real, complex industrial cases provided by our industrial partners (desensitized problems across production changeovers, parallel processing, repair/assembly, hydrogen/chemical-energy planning, network security, and power-unit commitment).

Each workspace has 5–16 linked CSV files (39–2,077 rows), averaging 2,787 variables (499–9,373) and 3,811 constraints (352–15,193):

Solver backendModelBUILD Pass@1 (%)REVISE (code) Pass@1 (%)EXPLAIN Rubric (0–100)
Gurobi 13.0.1gpt-5.545.055.086.19
deepseek-v4-pro10.045.076.43
deepseek-v4-flash0.055.078.66
COPT 8.1.0gpt-5.530.040.087.47
deepseek-v4-pro20.010.074.37
deepseek-v4-flash0.00.078.02

BUILD drops sharply when reconstructing multi-period couplings across 5–16 tables from scratch, confirming that our pipeline extends cleanly beyond IndustryOR. We will open-source HARD20 in a subsequent release.


[W1, Q1] Controlled Filesystem vs. Flat Comparison

In Table 3 (Appendix A.2, Table 7), we fix instances, requirements, data, solver, and scoring across six models, varying only inline prompt layout (Flat) vs. separate workspace files (Filesystem):

ModelBUILD FsBUILD FlatBUILD Δ\DeltaREVISE FsREVISE FlatREVISE Δ\DeltaEXPLAIN FsEXPLAIN FlatEXPLAIN Δ\Delta
gemini-3.1-pro727208794-769.1370.55-1.42
deepseek-v4-pro6770-38085-573.6273.76-0.14
gpt-5.55966-78589-485.0984.17+0.92
claude-sonnet-4.55362-98285-381.1179.68+1.43
qwen3-max4960-116875-772.9574.19-1.24
gpt-4o2544-193650-1456.1754.57+1.60
Average54.262.3-8.273.079.7-6.773.0172.82+0.2
  • Task difficulty in Flat: Even without file boundaries, the top model reaches 72% on BUILD (62.3% mean) and EXPLAIN averages 72.82, showing that business specifications, code REVISE, and solver-backed EXPLAIN add substantial difficulty beyond the seed problem.

  • Multi-file difficulty in Filesystem: Separate files lower mean Pass@1 by 8.2 pp on BUILD (0 to -19 pp) and 6.7 pp on REVISE (all six models drop), quantifying cross-file schema alignment and data loading challenges.

AuthorsOpenReview ↗

Author Response to Reviewer Twsh (2/3): Formulation Correctness & Requirement Coverage [W2]

[W2] Formulation Correctness and Implementing All Stated Requirements

1. Evaluation boundary and the open problem of formulation equivalence.

We agree with your observation: on a single input, omitting an inactive constraint, overlooking a newly added REVISE requirement, or altering variable definitions can occasionally yield the same optimal objective. We state this boundary explicitly in Section 4.1.

At the same time, automatically deciding whether two general LP/MILP formulations are mathematically equivalent remains an open problem, since one problem admits multiple valid formulations (e.g., variable substitutions, auxiliary variables, or Big-MM vs. indicator constraints) with no universal equivalence test.

Much as math and coding benchmarks (e.g., MATH, SWE-bench) evaluate final answers or test execution rather than non-unique intermediate derivations, solver-based Pass@1 remains the standard primary metric across OR benchmarks (ComplexOR, OptiMUS, OptiBench, IndustryOR, MIPLIB-NL).

2. Automated requirement-coverage check (SAC-Opt and Opt-Verifier, Appendix A.7, Tables 12–13).

To check requirement coverage directly, we adapted two ICML 2026 methods on 20 frozen instances per model for Gemini 3.1 Pro and Claude Opus 4.6 (40 outputs; gpt-5.6-sol, τa=0.7\tau_a = 0.7):

  • SAC-Opt (Zhang et al., 2026): checks whether code implements required variables, objectives, constraints, and parameters from the written specification (including updated rules).

  • Opt-Verifier (Liu et al., 2026): scores model structure separately from solution validity.

Candidate modelNNObjective / executionSAC-Opt anchor meanSAC-Opt anchor passOpt-Verifier structureOpt-Verifier solutionOpt-Verifier joint
Gemini 3.1 Pro2015/20 (75.0%)0.857217/20 (85.0%)7/20 (35.0%)9/20 (45.0%)4/20 (20.0%)
Claude Opus 4.62013/20 (65.0%)0.811915/20 (75.0%)6/20 (30.0%)9/20 (45.0%)3/20 (15.0%)
All audited cases4028/40 (70.0%)0.834632/40 (80.0%)13/40 (32.5%)18/40 (45.0%)7/40 (17.5%)

3. Manual code audit by two OR experts (Appendix A.7, Table 14).

Because general formulation equivalence is an open problem and our reference code is only one valid formulation among many, strict single-reference matching (32.5% pass; 11/40 solution-correct / model-inconsistent in Table 13) under-scores valid equivalent implementations.

We therefore asked two OR PhD researchers to manually audit all 159 passing submissions of Gemini 3.1 Pro (72 BUILD + 87 REVISE), checking line by line whether each program implemented every required variable, original constraint, and newly added REVISE requirement:

StageObjective-pass submissionsExpert A false positivesExpert B false positivesMean false-positive rate
BUILD722/72 (2.8%)3/72 (4.2%)3.5%
REVISE874/87 (4.6%)4/87 (4.6%)4.6%
Combined1596/159 (3.8%)7/159 (4.4%)4.1%

Both the 80.0% requirement-element pass rate under SAC-Opt and the low manual false-positive rate (4.1% overall, and 4.6% (4/87) on REVISE) show that passing submissions rarely omit stated or newly added requirements. Moreover, every REVISE task changes the optimal objective relative to BUILD (Section 3.2), so an unedited model cannot pass.

AuthorsOpenReview ↗

Author Response to Reviewer Twsh (3/3): Mitigating Data-Contamination Risk [Q2]

[Q2] Mitigating Data-Contamination Risk from Public Seeds

Potential pretraining exposure is an open challenge for any public benchmark and cannot be ruled out completely. OR-Space mitigates this risk in three ways:

  • 1. Non-public partner cases (HARD20, Appendix A.8): HARD20 builds 20 workspaces from desensitized real industrial cases provided by our enterprise partners rather than public benchmarks, offering an unexposed test set where accuracy drops markedly.

  • 2. Multi-file workspace transformation: Even when starting from a public seed, OR-Space converts it into linked CSVs, existing solver code, newly authored REVISE requirements, and solver logs (REVISE and EXPLAIN do not exist in IndustryOR). Solving them requires cross-table key resolution, code edits under new rules, and solver-grounded explanations.

  • 3. Versioned benchmark renewal (Appendix E.1): Because OR-Space uses a reusable four-step pipeline and ongoing partner collaborations rather than a fixed problem list, we can release new benchmark versions as older public instances become exposed—keeping each release fixed for fair comparison while supporting on-demand generation for private evaluations.


We hope these additional experiments and clarifications address your questions, and we welcome any further discussion.

Thanks for the detailed response. The "manual code audit by two OR experts" provides direct evidence of the evaluation's reliability and addresses my previous concerns. The gap between the results of automated verification and the manual audit still needs more explanation in the final manuscript, and the existing evaluation metrics and their limitations should be defined more clearly. Overall, the authors have addressed my primary concerns. I will maintain my initial positive rating.

Reviewer_ZzDJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OR-Space, a benchmark that expands 100100 IndustryOR problems into 300300 workspace tasks covering optimization model construction (BUILD), revision under changing requirements (REVISE), and explanation grounded in solver evidence (EXPLAIN). Its main contribution is the integration of these tasks with corresponding workspaces and scoring protocols to assess aspects of modeling across files and model maintenance that text-based evaluations do not fully capture. Experiments with 1616 models, two additional agent harnesses, and 44 solvers examine the effects of filesystem interfaces, code context, and solver choice, providing diagnostic insights into LLM capabilities and limitations in optimization modeling workflows.

Critical strengths

The paper addresses practically relevant aspects of optimization modeling through a clear evaluation design. Figures 1 to 3 specify task inputs and visibility: REVISE provides a correct prior implementation, while EXPLAIN provides validated original and revised workspaces, helping isolate the intended capabilities and avoid confounding from upstream errors. The experiments are informative: Table 3 reports average decreases of 8.28.2 and 6.76.7 percentage points for BUILD and REVISE with the filesystem interface, and Tables 4 to 9 examine interactions between code context and solver APIs. The evaluation includes recent models and the OpenCode/Pi harnesses. Supplementary validation includes calibration of 9090 responses against two OR experts and assessments of 3030 workspaces by 88 practitioners. The authors also acknowledge missed errors by the judge and limits to industrial representativeness, helping clarify the scope of the findings.

Critical weaknesses

The main concern is that the scoring protocol does not fully establish model correctness. Equation (2) requires successful execution and an objective value close to the reference, leaving open the possibility of omitted nonbinding constraints, incorrect feasible regions, or coincidentally matching objectives. Preservation of existing logic in REVISE also needs hidden parameter variants or constraint-level regression checks; independent verification of benchmark references does not validate submitted models. Evidence for industrial applicability remains limited: Figure 4 reports median variable and constraint counts of 66 each for BUILD and 1515 and 1616, respectively, for REVISE, with all workspaces synthesized from one public seed set. Practitioner feedback in Table 13 supports the task format but does not establish applicability to real project scales, imperfect data, or naturally occurring requirements. Because REVISE and EXPLAIN receive correct prerequisite implementations, Table 2 also measures individual capabilities without establishing success across the complete BUIL、REVISE、EXPLAIN workflow. The contribution needs clearer positioning against MIPLIB-NL and ORAgentBench, , particularly regarding model-data separation, execution across multiple files, and hidden feasibility and solution-quality checks. MIPLIB-NL is cited but absent from Table 1; jointly evaluating the three tasks may remain a useful distinction. In addition, the judge calibration in Table 12, with an MAE of 6.96.9 against expert scores and detection of 18/24=75%18/24=75\% of substantive errors, does not by itself establish the stability of EXPLAIN rankings across all 1616 models. The efficiency metric in Figure 6, which equally averages Pass@1 and rubric scores before dividing by tokens, is descriptive and does not measure end-to-end latency or cost; Appendix A.5 also leaves the coverage of the token-accounting subset unspecified. Finally, the evaluation tolerance ϵ\epsilon in Equation (2) should be reported, given its explicit distinction from the construction tolerance of 10−310^{-3}, and Figure 6 should clarify its metric definition and token accounting.

Questions for authors

see weakness

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer ZzDJ (1/5): Overview & Automated Formulation Audit [W1]

We thank Reviewer ZzDJ for the detailed and constructive feedback. Following your suggestions, we added five empirical studies and updated the manuscript:

  • Formulation & REVISE code audit (Appendix A.7): automated checks (SAC-Opt, Opt-Verifier) plus a line-by-line manual audit by two OR experts.
  • Larger industrial cases (HARD20, Appendix A.8): 20 desensitized real, complex industrial cases from our industrial partners.
  • Sequential BUILD → REVISE → EXPLAIN (Appendix A.11): each stage runs on the model's own output from the preceding stage.
  • Fault diagnosis and repair (Appendix A.10): 20 REVISE workspaces seeded with runtime and modeling faults.
  • Alternative EXPLAIN judges (Appendix F.6): re-scoring all 16 models, plus separated cost/token metrics (Figure 6, Appendix A.5), tolerances (Appendix A.9), and benchmark comparisons (Section 2, Table 1, Appendix D).

[W1] Formulation Correctness and Automated Verification

1. Evaluation boundary and the open problem of formulation equivalence.

We agree that matching the reference optimal value on one supplied instance confirms executable solving on that input, rather than guaranteeing every nonbinding constraint or feasible-region detail (nor does offline validation of our reference code establish a candidate model's structure). We now state this boundary explicitly around Equation (2).

At the same time, automatically deciding whether two general LP/MILP formulations are mathematically equivalent remains an open problem, as one problem admits many equivalent formulations (e.g., variable substitutions, auxiliary variables, or different constraint splits).

Much like outcome-based evaluation in math reasoning (GSM8K, MATH) and software engineering (SWE-bench)—where valid intermediate steps or implementations are non-unique—solver execution paired with optimal-objective agreement remains the standard primary metric across OR benchmarks (ComplexOR, OptiMUS, OptiBench, IndustryOR, MIPLIB-NL).

2. Automated formulation check (SAC-Opt and Opt-Verifier, Appendix A.7, Tables 12–13).

To inspect code structure directly, we adapted two ICML 2026 methods using gpt-5.6-sol (τa=0.7\tau_a = 0.7) on 20 frozen instances per model (40 outputs across Gemini 3.1 Pro and Claude Opus 4.6):

  • SAC-Opt (Zhang et al., 2026): checks correspondence on required variables, objectives, constraints, and parameters.
  • Opt-Verifier (Liu et al., 2026): evaluates model structure separately from solution validity.
Candidate modelNNObjective / executionSAC-Opt anchor meanSAC-Opt anchor passOpt-Verifier structureOpt-Verifier solutionOpt-Verifier joint
Gemini 3.1 Pro2015/20 (75.0%)0.857217/20 (85.0%)7/20 (35.0%)9/20 (45.0%)4/20 (20.0%)
Claude Opus 4.62013/20 (65.0%)0.811915/20 (75.0%)6/20 (30.0%)9/20 (45.0%)3/20 (15.0%)
All audited cases4028/40 (70.0%)0.834632/40 (80.0%)13/40 (32.5%)18/40 (45.0%)7/40 (17.5%)
Verifier breakdownGemini 3.1 ProClaude Opus 4.6All
Solution-correct / model-inconsistent5611
Solution-correct / model-consistent437
Solution-incorrect / model-consistent224
Both inconsistent8816
Undetermined / parse failure112
Total audited instances202040

Here, 28/40 (70.0%) pass the objective check and 32/40 (80.0%) cover required elements under SAC-Opt, whereas only 13/40 (32.5%) pass Opt-Verifier's structure check (11/40 are solution-correct / model-inconsistent).

Because general formulation equivalence is an open problem and our reference model is only one valid formulation among many, grading against a single reference penalizes valid equivalent formulations—making rubric-based structural verifiers less reliable as a primary metric than solver-grounded objective checks.

AuthorsOpenReview ↗

Author Response to Reviewer ZzDJ (2/5): Manual REVISE Audit [W1] & Industrial HARD20 [W2]

[W1] Preserving Prior Logic in REVISE (Manual Audit)

3. Manual audit of REVISE logic preservation (Appendix A.7, Table 14).

To verify directly whether passing REVISE programs keep valid prior logic rather than matching the updated objective by coincidence, two OR PhD researchers manually audited all 87 passing REVISE programs (and all 72 passing BUILD programs) of Gemini 3.1 Pro line by line.

They checked for dropped prior constraints, altered variable types or bounds, and lost CSV parameter bindings, without penalizing equivalent rewrites. Both experts flagged only 4/87 (4.6%) passing REVISE programs (4.1% across all 159 passes; Table 14), confirming that over 95% of passing REVISE runs preserve valid prior logic.

4. Primary metric and supplementary checks.

Taken together (4.1% manual false-positive rate vs. 11/40 valid-solution rejections under strict structure matching), solver-based Pass@1 provides a more faithful primary score across all 16 models than single-reference structural rubrics, complemented by the automated and manual audits in Appendix A.7.


[W2] Larger Industrial Cases (HARD20) and Imperfect Workspaces

To evaluate OR-Space beyond IndustryOR, we built HARD20, a separate test set of 20 real, complex industrial cases from our industrial partners (Appendix A.8, Table 16).

HARD20 is not created by scaling up IndustryOR instances. It comes from an independent enterprise source spanning production changeovers, parallel processing, repair and assembly, hydrogen/chemical-energy planning, network security, and power-unit commitment, with proprietary numbers desensitized while preserving business rules and cross-table dependencies.

1. Industrial scale and pipeline reuse.

Each HARD20 workspace contains 5–16 linked CSV files (39–2,077 rows), and reference formulations average 2,787 variables and 3,811 constraints (up to 9,373 variables and 15,193 constraints). All 20 workspaces were built with the exact same four-step pipeline, without modifying task interfaces or scoring. We will open-source HARD20 in a subsequent release.

2. Evaluation results on HARD20 (Gurobi 13.0.1 and COPT 8.1.0).

SolverModelBUILD (%)REVISE (%)EXPLAIN
GurobiGPT-5.545.055.086.19
DeepSeek V4 Pro10.045.076.43
DeepSeek V4 Flash0.055.078.66
COPTGPT-5.530.040.087.47
DeepSeek V4 Pro20.010.074.37
DeepSeek V4 Flash0.00.078.02

Accuracy drops sharply relative to the 100-instance benchmark (under Gurobi, GPT-5.5 drops from 59% → 45% on BUILD and 85% → 55% on REVISE; DeepSeek V4 Pro drops from 67% → 10% on BUILD). More broadly, problem scale is a property of the input seed, whereas OR-Space establishes the multi-file BUILD–REVISE–EXPLAIN paradigm and a reusable pipeline that scales directly as larger enterprise seeds are supplied.

3. Imperfect data and faulty code.

Beyond scale, Appendix A.6 analyzes semi-structured CSVs and case-sensitive key mismatches, and Appendix A.10 evaluates repairing broken codebases (in Post 3 below).

AuthorsOpenReview ↗

Author Response to Reviewer ZzDJ (3/5): Practitioner Study, Sequential Workflow & Fault Repair [W1, W2]

[W2] Practitioner Representativeness Study (Table 22)

In Section 3.3 and Appendix F.5, we clarify the scope of our practitioner study (Table 22, original Table 13). Eight industrial OR practitioners (K=8K=8, each with ≥3\ge 3 years of LP/MILP experience and no involvement in benchmark construction) independently rated 30 stratified workspaces (M=30M=30: 10 per task) on a 5-point scale:

Item (K=8K=8 raters, M=30M=30 workspaces, 5-point scale)Mean (SD)Ratings ≥4\ge 4
Requirement documents resemble real business specifications4.1 (0.7)78%
Workspace organization resembles a real project layout3.9 (0.8)70%
Revisions resemble realistic change requests4.0 (0.8)73%
Tasks are likely to arise in industrial practice3.9 (0.7)71%
Solving them requires skills used in practice4.3 (0.6)85%
Overall4.0 (0.8)75%

Ratings average 4.0 overall (75% >= 4, Krippendorff's α=0.64\alpha = 0.64), led by practical modeling skills (4.3) and specification realism (4.1). As stated in Section 3.3 and Appendix F.5, these ratings support representativeness in business documents, change requests, and modeling skills, whereas enterprise model scale is evaluated separately on HARD20 (Appendix A.8).


[W2] Sequential BUILD → REVISE → EXPLAIN Evaluation

Table 2 evaluates BUILD, REVISE, and EXPLAIN separately on verified inputs to isolate per-task capability. To complement this, we ran a sequential BUILD → REVISE → EXPLAIN evaluation across all 100 instances and all 16 models (1,600 runs; Appendix A.11, Tables 18–19).

In this sequential setting, each model revises the code it generated itself in BUILD (rather than reference code), and then explains the solver outputs from that same run (requiring EXPLAIN > 70 for full completion).

Errors compound across stages: out of 1,600 runs, 769 (48.06%) pass BUILD, 522 (32.63%) pass both BUILD and REVISE (67.88% conditional), and 299 (18.69%) complete all three stages (57.28% conditional). Rankings also shift: Gemini 3.1 Pro leads standalone BUILD (72%), whereas GPT-5.5 leads full three-stage completion (45%). Standalone scores thus isolate each capability, while sequential runs measure reliability across dependent steps.


[W1, W2] Fault Diagnosis and Repair in REVISE

We extended REVISE with a fault-repair study (Appendix A.10, Table 17) on 20 faulty workspaces per model (10 runtime faults that fail to execute and 10 modeling faults that run an incorrect formulation) across Gemini 3.1 Pro and GPT-5.5 (40 runs total):

Fault TypeFault IdentifiedExecutableObjective MatchedConstraints KeptAll Four Passed
Runtime (20)90.0%95.0%80.0%30.0%25.0%
Modeling (20)90.0%95.0%95.0%30.0%30.0%
Overall (40)90.0%95.0%87.5%30.0%27.5%

While 95.0% of repaired programs execute and 87.5% recover the target objective, only 30.0% pass the static check for preserving original constraints and 27.5% pass all four checks (Appendix A.10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.