- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OR-Space, a benchmark for evaluating LLM agents on industrial optimization tasks. Instead of presenting each problem as a self-contained description, OR-Space organizes the required information into executable workspaces containing business documents, structured data, code, and solver records.
The benchmark consists of three independently evaluated tasks: BUILD, REVISE, and EXPLAIN. It contains 300 task instances derived from 100 IndustryOR problems. The authors evaluate 16 language models and two agent harnesses, and study the effects of workspace organization, revision context, and solver backend.
Critical strengths
-
The benchmark addresses a relevant gap in existing OR evaluations. Building an optimization model from business requirements and structured data, updating an existing implementation, and explaining solver results are different capabilities. Evaluating them separately is useful, particularly since many existing benchmarks focus primarily on initial model construction.
-
The experimental analysis is reasonably comprehensive. The filesystem-versus-flat comparison is particularly interesting, showing that organizing the same information into separate files can make model construction more difficult. The comparisons across revision contexts and solver backends are also useful. The authors have made a substantial effort to validate the benchmark, including reference-model verification, failure analysis, expert calibration of the EXPLAIN evaluator, and practitioner assessment of task representativeness.
Critical weaknesses
- The BUILD and REVISE evaluation metric does not fully capture formulation correctness.
According to Section 4.1, a submission is considered successful when the generated program executes with the required solver and its objective value agrees with the reference optimum within a specified tolerance.
However, matching the optimal objective value does not necessarily mean that the generated mathematical model is correct. An agent may omit a constraint that is inactive at the reference optimum, or implement an incorrect feasible region, while still obtaining the expected objective value. This is particularly relevant for REVISE, where preserving valid modeling logic is explicitly part of the task.
The authors carefully validate the reference implementations during dataset construction. However, this does not establish the correctness of agent-generated models that pass the objective-based evaluation.
I would like to see some analysis of this potential false-positive issue, perhaps through manual inspection of passing submissions or testing generated models under modified parameter values. This would help establish how reliably Pass@1 reflects actual modeling capability.
- The EXPLAIN evaluator is reasonably designed, but its validation does not yet establish the reliability of model rankings.
I appreciate the expert calibration study in Appendix F. The comparison with human experts is useful and provides more evidence than an unvalidated LLM-as-a-judge setup.
However, the study covers 90 responses from only three models. The judge achieves a total-score MAE of 6.9 points and identifies 18 of the 24 responses containing substantive mathematical errors. These results are informative, but they do not establish whether the relative rankings remain stable across all 16 evaluated models.
I agree with the authors' own observation in Appendix F.4 that the calibration results are insufficient to validate the full ranking. Further analysis would strengthen the interpretation of the EXPLAIN results, especially for models with similar scores.
Questions for authors
-
Have the authors examined whether any BUILD or REVISE submissions that pass the objective-based metric contain incorrect mathematical formulations? A manual audit of a sample of passing submissions, or evaluation under perturbed input parameters, would be particularly helpful.
-
How robust are the EXPLAIN model rankings to the choice of evaluator? Would a different judge or additional expert evaluations produce similar rankings, especially among models with relatively close scores?
2 comments
Author Response to Reviewer kXXA (2/2): EXPLAIN Rubric & Multi-Judge Consistency [W2, Q2]
[W2, Q2] EXPLAIN Rubric Design and Ranking Consistency Across Judge Models
1. Rubric structure (Appendix C, Table 20).
Our EXPLAIN rubric combines programmatic checks with structured criteria:
-
Deterministic
Exact Coverage(35/100pts): scored directly by code—matching required numbers, variable names, and CSV columns after text normalization, without calling an LLM judge. -
Checklist & anchored criteria (
65/100pts): per-question checklist items (Reasoning,35pts) and explicit0–3scoring rules (Grounding,20pts;Answer Quality,10pts). -
Unsupported-claim penalties (
0 / 5 / 12 / 20pts): require quoting the specific unsupported statement (0penalty if none is quoted).
2. Re-scoring all 16 models with alternative judges (Appendix F.6, Table 23).
Keeping the rubric, prompts, and temperature fixed, we re-scored all 100 EXPLAIN instances across all 16 models using two non-evaluated models (gpt-5.4, gpt-5.6-sol), gemini-3.1-pro (testing self-preference), and a three-judge mean (gpt-5.1, gpt-5.4, gpt-5.6-sol):
| Judge configuration | Spearman | Kendall | Top-5 overlap | Pairwise order (120 pairs) |
|---|---|---|---|---|
gpt-5.4 | 0.99 | 0.93 | 5/5 | 116/120 (96.7%) |
gpt-5.6-sol | 0.98 | 0.90 | 4/5 | 114/120 (95.0%) |
gemini-3.1-pro | 0.98 | 0.88 | 4/5 | 113/120 (94.2%) |
| Three-judge mean | 0.99 | 0.97 | 5/5 | 118/120 (98.3%) |
Rankings remain highly consistent across judges (Spearman –; pairwise order agreement 94.2%–98.3%). Paired bootstrap 95% CIs include zero for the 8 close pairs (< 1.5 pts apart).
Top-5 overlap is 4/5 or 5/5. On the 90-response expert set, gpt-5.4 (MAE = 6.5) and gpt-5.6-sol (7.1) closely track gpt-5.1 (6.9), and gemini-3.1-pro shifts its own mean by only +0.35 (rank 8 unchanged). Accordingly, we treat models within 2 points as tied (Section 4.1, Appendix F.6).
We hope these audits and supplementary results address your questions, and we welcome any further discussion.
Author Response to Reviewer kXXA (1/2): Formulation Correctness & Manual Audit [W1, Q1]
We thank Reviewer kXXA for the thoughtful and constructive feedback. In response to your two questions, we:
Asked two OR experts to manually audit every passing
BUILDandREVISEsubmission ofGemini 3.1 Pro, alongside an automated formulation check (Appendix A.7).Re-scored all 16 models on
EXPLAINwith alternative judge models (Appendix CandAppendix F.6).[W1, Q1] Formulation Correctness and Manual Audit of Passing Submissions
Evaluation boundary and the open problem of formulation equivalence.
We agree that matching the reference optimal objective on one input can occasionally pass a flawed formulation (e.g., omitting a nonbinding constraint or relaxing integer variables). We state this boundary explicitly in Section 4.1 and followed your suggestion to inspect passing submissions directly (Appendix A.7).
At the same time, automatically deciding whether two general LP/MILP formulations are mathematically equivalent remains an open problem, as one problem admits multiple equivalent formulations (e.g., variable substitutions, auxiliary variables, Big-M vs. indicator constraints, or different constraint splits).
Much as math and coding benchmarks (GSM8K/MATH, SWE-bench) grade final answers or test outcomes rather than non-unique intermediate steps, solver execution with optimal-objective agreement remains the standard primary metric across OR benchmarks (ComplexOR, OptiMUS, OptiBench, IndustryOR, MIPLIB-NL), while recent verifiers (EquivaMap, ORGEval, SAC-Opt, Opt-Verifier) still lack general equivalence guarantees across multi-file solver code.
1. Complete manual audit by two OR experts (
Appendix A.7, Table 14).Two OR PhD researchers manually audited every
Gemini 3.1 Prosubmission that passed our objective metric across the 100-instance benchmark—covering all 72BUILDpasses and all 87REVISEpasses (159programs;318independent reviews).A program was flagged as a false positive if it omitted a required constraint, changed variable types or bounds, altered the feasible region, dropped an input parameter binding, or broke valid prior logic in
REVISE(equivalent reformulations and helper variables were not penalized):BUILDREVISEAcross all
318checks,13false-positive flags occurred (4.1%mean:3.5%onBUILDand4.6%onREVISE). While objective-only false positives do occur, their rate onGemini 3.1 Prois low (~4.1%); over95%of passing submissions implement valid formulations (Appendix A.7).2. Automated formulation check (
SAC-OptandOpt-Verifier,Appendix A.7, Tables 12–13).We also adapted two ICML 2026 methods using
gpt-5.6-sol(τa=0.7) on 20 frozen instances per model (40outputs acrossGemini 3.1 ProandClaude Opus 4.6):SAC-Opt (Zhang et al., 2026): checks whether code covers the stated variables, objectives, constraints, and parameters.
Opt-Verifier (Liu et al., 2026): evaluates model structure separately from solution validity.
SAC-Opt confirms high element coverage (
80.0%), whereas strict single-reference structure matching is32.5%(11/40solution-correct / model-inconsistentcases inTable 13).Because general formulation equivalence is an open problem and our reference model is only one valid representation among many, single-reference verifiers penalize valid equivalent formulations (
11/40valid-solution rejections vs.4.1%manual false positives)—making solver-groundedPass@1a more direct primary metric (Section 4.1, Appendix A.7).