- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces ORC-Bench, a benchmark designed to evaluate the capabilities of Large Language Models (LLMs) in optimization and reasoning under interacting structural, physical, and operational constraints. The benchmark comprises 30 tasks across ten categories, organized into three difficulty levels (Small, Medium, and Large), and covers two structured input representations: graphs and tables. The tasks span power-grid analysis and optimization, as well as tabular problems involving credit scoring, phishing detection, and botnet detection. Collectively, they aim to assess six capabilities: graph understanding, tabular understanding, mathematical formulation, arithmetic execution, problem abstraction, and multi-step optimization.
The main contribution is a constraint-aware evaluation framework that goes beyond conventional accuracy-based metrics by incorporating domain-specific and physics-informed checkers. These assess structured-output validity, numerical accuracy, and compliance with physical and logical constraints. The authors evaluate nine LLMs across different model families, sizes, and reasoning configurations, examining the effects of model scaling and explicit reasoning.
The experimental results indicate that, while larger models and reasoning-enhanced variants generally improve performance on several tasks, even strong models struggle with complex, jointly coupled constraints, particularly in optimal power flow. The authors further investigate these findings through failure-mode analysis, prompting and in-context learning experiments, calibration analysis, fine-tuning, distribution-shift robustness, and limited tool-assisted evaluations. Overall, the paper highlights a gap between conventional LLM reasoning performance and the ability to generate feasible solutions to constrained optimization problems.
Critical strengths
The paper is well motivated and positioned within the existing literature. The authors identify an important gap in evaluating LLMs' ability to reason and optimize under interacting structural, physical, and operational constraints. The Introduction provides a compelling motivation, and the related-work discussion clearly distinguishes ORC-Bench from existing mathematical reasoning, constraint-satisfaction, and optimization-modeling benchmarks. This establishes the significance of the problem and makes the proposed benchmark a relevant contribution to LLM evaluation.
The benchmark offers a meaningful and diverse evaluation framework grounded in real-world applications. ORC-Bench covers ten task categories spanning graph reasoning, numerical computation, optimization, and constraint discovery, using established power-grid models and tabular datasets from finance and cybersecurity. A particularly valuable aspect is the incorporation of physics-informed and domain-specific constraint checkers that evaluate feasibility and logical consistency rather than relying solely on answer accuracy. This makes the benchmark useful for identifying limitations that conventional reasoning evaluations may overlook.
The paper provides an extensive empirical evaluation with interesting findings and useful failure analysis. The authors evaluate LLMs across different model families, sizes, and reasoning configurations, examining the effects of model scaling and explicit reasoning. The results reveal notable differences between simpler reasoning tasks and problems involving coupled constraints, particularly in optimal power flow. The qualitative failure analysis further identifies recurring errors in graph interpretation, power-flow reasoning, and constraint handling. Together, these experiments provide useful insights into the strengths and limitations of current LLMs and strengthen the practical value of the benchmark.
Critical weaknesses
Some task definitions and evaluation criteria raise concerns about whether the benchmark accurately measures the intended capabilities. In Task 4 (Contingency Analysis), the ground truth is determined by the largest absolute change in power flow, which does not necessarily correspond to the most overloaded transmission line relative to its capacity. Similarly, Task 10 (Constraint Expression Recovery) evaluates predictions against a specific reference expression, although the authors acknowledge that 93–95% of rejected expressions satisfy all the displayed rows. Furthermore, Task 6 (Optimal Power Flow) primarily reports the proportion of individual constraints satisfied, rather than the percentage of complete solutions satisfying all constraints simultaneously. These issues raise questions about the interpretation of benchmark scores and the extent to which they reflect successful constrained reasoning and optimization.
The experimental evaluation lacks a human or domain-expert baseline to contextualize the difficulty of the proposed tasks. The authors emphasize that even frontier LLMs struggle on ORC-Bench, but it is unclear how the observed performance compares with human experts, particularly on complex optimization tasks that ordinarily require computational tools. In addition, some results may be influenced by output formatting and parsing rather than reasoning capabilities alone. The authors acknowledge sensitivity to regex-based parsing, and several models exhibit substantial formatting failures. A clearer separation between reasoning errors and output-format violations, alongside suitable human or expert-assisted baselines, would make the conclusions about benchmark difficulty more convincing.
The definition of task difficulty is not consistently controlled across the benchmark. While the Small, Medium, and Large variants of graph-based tasks correspond to increasing network sizes (14, 30, and 118 buses), the corresponding variants of tabular tasks use entirely different datasets and application domains. Consequently, changes in model performance across these variants cannot be attributed solely to increasing problem complexity, as dataset characteristics, feature dimensionality, and underlying constraints also differ. This limits the interpretation of difficulty-related comparisons and the generalizability of conclusions about how LLM performance scales with task complexity.
Questions for authors
In Task 4, why is the largest change in power flow used to identify an overloaded line, rather than considering line capacity? For Task 10, how do the authors justify rejecting alternative expressions that satisfy all displayed rows? Would different evaluation criteria significantly affect the results?
In Task 6, the evaluation reports the proportion of individual constraints satisfied. Could the authors also report the percentage of complete solutions satisfying all constraints simultaneously?
How do the authors contextualize benchmark difficulty without human or domain-expert baselines, particularly for optimization problems that typically require computational solvers?
What justifies the Small, Medium, and Large difficulty ordering for tabular tasks when they involve different datasets and domains?
3 comments
R3W2.1, R3Q3
it is unclear how the observed performance compares with human experts, particularly on complex optimization tasks that ordinarily require computational tools
All the grid tasks of our benchmarks are real, and routinely run by grid operators using traditional computations tools for large grids. Tasks 1-5 can all be run without optimization tools.
-
Task 1 (path finding) & 2 (connectivity): Operators constantly need to know how power can physically reach a given area when lines are on outage (planned maintenance or faults). Checking whether parts of the network are energized or islanded is a continuous background function of a grid operator. After a trip, the system must detect islands (and flag dead buses) before any power flow can be solved. With a single-line diagram, an operator can trace a path or confirm connectivity in seconds, even with 1-2 lines removed for all the grids of our study (up to 118 bus). Note this is exactly why our paper’s small models failing here is notable: it is trivial for a human.
-
Task 3 (DC power flow): We restricted the evaluation to a DC linear flow. The DC model reduces to a linear system. For 14 buses you remove the slack bus and solve a 13x13 system, B'θ = P, then compute each line flow as (θi − θj)/x. With a calculator and a spreadsheet, it is feasible, but can take hours for manual calculation.
-
Task 4 (N-1 contingency): Power grid operators generate PTDF tables it becomes a few multiplications per line, so an operator could easily solve Task 4 for one contingency. PTDF (Power Transfer Distribution Factors) and LODF tables are precomputed sensitivity matrices derived from the DC power flow model (Task 3).
-
Task 5 (grid control). This is rule-based: read the frequency and voltage, match them to a playbook entry, and pick the action. It needs judgment and procedure knowledge, not computation, and it is precisely what operators are trained on.
-
Task 6 (OPF). The AC problem is non-linear and cannot be exactly solved without an optimlization algorithm. A human can only approximate it, for example by merit-order dispatch (cheapest generator first) and then checking limits, which gives a feasible but suboptimal answer.
We summarize below which tasks can be done by human operators:
| Task | Manual | Realistic method |
|---|---|---|
| 1 Path finding | Yes, easy | Trace on the single-line diagram |
| 2 Connectivity | Yes, easy | Trace on the single-line diagram |
| 3 DC power flow | Yes, slowly | Spreadsheet and calculator |
| 4 Contingency | Yes, slowly | PTDF/LODF tables |
| 5 Power grid control | Yes, easy | Playbook or operator judgment |
| 6 OPF | No | Merit-order approximation only, or optimization solvers |
R3W2.2
some results may be influenced by output formatting and parsing rather than reasoning capabilities alone
For every task, the output formatting is specified and the LLM should respect it.
Task 5 for instance requires the answer to be formatted as [[A]].
A quick experiment asking for a JSON response instead ({"answer": "A"}, also a common format for LLMs) shows that this required formatting doesn't influence the scores:
| Model | Task 5 | Task 5 - different formatting |
|---|---|---|
| Qwen3 8B | 71.4% | 70.8% |
| Qwen2.5 32B | 83.0% | 81.3% |
R3W3.1, R3Q4
While the Small, Medium, and Large variants of graph-based tasks correspond to increasing network sizes (14, 30, and 118 buses), the corresponding variants of tabular tasks use entirely different datasets and application domains.
The Small, Medium, and Large benchmarks in the tabular setting reflect the classification and raking of these datasets in the original TabularBench benchmark.
In their original paper, they proposed datasets with increasing difficulty, based on the number and complexity of constraints.
URL (our S variant) is the simplest use-case with only 7 linear feature relationship constraints and 7 boolean single-feature constraints. Next is LCLD (our M variant) that has 19 single-feature immutability constraints, 3 linear and 7 non-linear feature relationship constraints. Finally, CTU, (our L variant) has 324 single-feature immutability constraints and 360 linear feature relationship constraints.
We agree with the reviewer that while the scale of constraints and relationship increases between the S, M, and L cases, the domain knowledge also chanes. Therefore, we propose to rephrase the tabular tasks as A, B, C rather than a classification by size Small, Medium, and Large.
Thank you for recognizing the importance of the gaps we are addressing, the value of our benchmark and the extensiveness pf pir empirical evaluation. We address in the following your remaining concerns.
R3W1.1, R3Q1.1
In the main paper, we defined Task 4 problem as "the model predicts which line overloads most"
In the appendix, we defined Task 4 problem as: "the model must identify the transmission line most likely to overload (experience the highest flow increase)"
We confirm that the formulation in the main paper is the correct one. Task 4 indeed consists at predicting which line is overloading (i.e., over line power boundaries), and not the one with highest increase of load.
We also confirm that all the results reported are correct both in the main paper and in appendix and reflect the proper formulation in the main paper.
We will update the appendix to correctly read: "the model must identify the transmission line most likely to overload (experience a power flow loading over the line flow boundaries). The standard line boundaries are defined in the problem as over 1.05 PU (105% nominal loads)."
R3W1.2, R3Q1.2
Thank you for raising this point. The objective of Task 10 was to check if the LLMs can recover one of the domain constraints established by the creators of the Tabularbench benchmark. However there was a confusion in the definition of the task in Appendix A.13.
Therefore we are now reporting for Task T10 two metrics:
Exact reconstruction: Whether the LLM recovered one of the human curated tasks, or any equivalent expression using the constraint parser of TabularBench. This is the metric our paper is currently reporting.
Validity: Whether the LLM recovered a constraint that is valid on the whole unseen test set of TabularBench.
URL dataset:
LCLD dataset:
CTU dataset:
The validity results are much higher than the exact reconstruction, given that the LLM can return simple constraints to pass this metric.
We analyzed already this phenomonon in Appendix B.1. Our analyses showed for URL three recurring patterns: 1) latching onto a column that happens to be constant in the sample (nb_colon == 1), 2) restating a variable’s obvious domain or boundaries (ip * (ip-1) == 0), and 3) tightening a bound with slack observed in the sample (length_url >= length_hostname + 7 in place of length_hostname <= length_url).
On the other hand, if an LLM achieves 100% exact constraint reconstructionn it may hint to LLM memorization from the TabularBench paper and their repository. Indeed, they have disclosed the exact formulation of each expert-designed constraint.
Thank you again for raising this discussion. The current paper only reports the exact reconstruction metric. If these additional results adress your concerns, we can revise the manuscript accordingly and update the formulation in appendix A13 and the results in Table 17 in appendix with both metrics.
R3W1.3, R3Q2
Currently no LLM manages to provide a single sample satisfying all constraints, accross all grid sizes. This makes the percentage of complete solutions satisfying all constraints simultaneously exactly 0% for all models and sizes.
We will include this 0% column in the revision.