acceptodds
Under review as a conference paper at ICLR 2027

ORC-Bench: Benchmarking LLMs Capabilities for Optimization and Reasoning Under Constraints

Abstract

Large Language Models (LLMs) have achieved great performance on a broad range of knowledge and reasoning benchmarks, yet existing evaluations rarely probe their ability to reason and optimize end-to-end under interacting structural, physical, and operational constraints. We introduce ORC-Bench, a benchmark for evaluating LLMs on Optimization and Reasoning under Constraints over realistic structured inputs. ORC-Bench comprises 30 tasks spanning ten categories, three difficulty levels, and two data modalities, jointly probing six skills: graph understanding, tabular understanding, mathematical formulation, arithmetic execution, problem abstraction, and multi-step optimization. Unlike existing benchmarks, ORC-Bench scores models with constraint-aware metrics derived from physics-informed and domain-specific checkers, going beyond accuracy to measure structured-output validity and constraint-satisfaction rates. Evaluating nine LLMs across architectures, scales, and reasoning regimes, we find that while scale and explicit reasoning improve performance on our tasks, all models (including frontier reasoning models with human performance on MATH-500 and ARC-AGI) struggle to satisfy interacting physical and logical constraints while optimizing domain specific objectives.

Primary area:
datasets and benchmarks
Keywords:
benchmark, large language models, reasoning, constraints, optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9sHVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces ORC-Bench, a benchmark designed to evaluate the capabilities of Large Language Models (LLMs) in optimization and reasoning under interacting structural, physical, and operational constraints. The benchmark comprises 30 tasks across ten categories, organized into three difficulty levels (Small, Medium, and Large), and covers two structured input representations: graphs and tables. The tasks span power-grid analysis and optimization, as well as tabular problems involving credit scoring, phishing detection, and botnet detection. Collectively, they aim to assess six capabilities: graph understanding, tabular understanding, mathematical formulation, arithmetic execution, problem abstraction, and multi-step optimization.

The main contribution is a constraint-aware evaluation framework that goes beyond conventional accuracy-based metrics by incorporating domain-specific and physics-informed checkers. These assess structured-output validity, numerical accuracy, and compliance with physical and logical constraints. The authors evaluate nine LLMs across different model families, sizes, and reasoning configurations, examining the effects of model scaling and explicit reasoning.

The experimental results indicate that, while larger models and reasoning-enhanced variants generally improve performance on several tasks, even strong models struggle with complex, jointly coupled constraints, particularly in optimal power flow. The authors further investigate these findings through failure-mode analysis, prompting and in-context learning experiments, calibration analysis, fine-tuning, distribution-shift robustness, and limited tool-assisted evaluations. Overall, the paper highlights a gap between conventional LLM reasoning performance and the ability to generate feasible solutions to constrained optimization problems.

Critical strengths

The paper is well motivated and positioned within the existing literature. The authors identify an important gap in evaluating LLMs' ability to reason and optimize under interacting structural, physical, and operational constraints. The Introduction provides a compelling motivation, and the related-work discussion clearly distinguishes ORC-Bench from existing mathematical reasoning, constraint-satisfaction, and optimization-modeling benchmarks. This establishes the significance of the problem and makes the proposed benchmark a relevant contribution to LLM evaluation.

The benchmark offers a meaningful and diverse evaluation framework grounded in real-world applications. ORC-Bench covers ten task categories spanning graph reasoning, numerical computation, optimization, and constraint discovery, using established power-grid models and tabular datasets from finance and cybersecurity. A particularly valuable aspect is the incorporation of physics-informed and domain-specific constraint checkers that evaluate feasibility and logical consistency rather than relying solely on answer accuracy. This makes the benchmark useful for identifying limitations that conventional reasoning evaluations may overlook.

The paper provides an extensive empirical evaluation with interesting findings and useful failure analysis. The authors evaluate LLMs across different model families, sizes, and reasoning configurations, examining the effects of model scaling and explicit reasoning. The results reveal notable differences between simpler reasoning tasks and problems involving coupled constraints, particularly in optimal power flow. The qualitative failure analysis further identifies recurring errors in graph interpretation, power-flow reasoning, and constraint handling. Together, these experiments provide useful insights into the strengths and limitations of current LLMs and strengthen the practical value of the benchmark.

Critical weaknesses

Some task definitions and evaluation criteria raise concerns about whether the benchmark accurately measures the intended capabilities. In Task 4 (Contingency Analysis), the ground truth is determined by the largest absolute change in power flow, which does not necessarily correspond to the most overloaded transmission line relative to its capacity. Similarly, Task 10 (Constraint Expression Recovery) evaluates predictions against a specific reference expression, although the authors acknowledge that 93–95% of rejected expressions satisfy all the displayed rows. Furthermore, Task 6 (Optimal Power Flow) primarily reports the proportion of individual constraints satisfied, rather than the percentage of complete solutions satisfying all constraints simultaneously. These issues raise questions about the interpretation of benchmark scores and the extent to which they reflect successful constrained reasoning and optimization.

The experimental evaluation lacks a human or domain-expert baseline to contextualize the difficulty of the proposed tasks. The authors emphasize that even frontier LLMs struggle on ORC-Bench, but it is unclear how the observed performance compares with human experts, particularly on complex optimization tasks that ordinarily require computational tools. In addition, some results may be influenced by output formatting and parsing rather than reasoning capabilities alone. The authors acknowledge sensitivity to regex-based parsing, and several models exhibit substantial formatting failures. A clearer separation between reasoning errors and output-format violations, alongside suitable human or expert-assisted baselines, would make the conclusions about benchmark difficulty more convincing.

The definition of task difficulty is not consistently controlled across the benchmark. While the Small, Medium, and Large variants of graph-based tasks correspond to increasing network sizes (14, 30, and 118 buses), the corresponding variants of tabular tasks use entirely different datasets and application domains. Consequently, changes in model performance across these variants cannot be attributed solely to increasing problem complexity, as dataset characteristics, feature dimensionality, and underlying constraints also differ. This limits the interpretation of difficulty-related comparisons and the generalizability of conclusions about how LLM performance scales with task complexity.

Questions for authors

In Task 4, why is the largest change in power flow used to identify an overloaded line, rather than considering line capacity? For Task 10, how do the authors justify rejecting alternative expressions that satisfy all displayed rows? Would different evaluation criteria significantly affect the results?

In Task 6, the evaluation reports the proportion of individual constraints satisfied. Could the authors also report the percentage of complete solutions satisfying all constraints simultaneously?

How do the authors contextualize benchmark difficulty without human or domain-expert baselines, particularly for optimization problems that typically require computational solvers?

What justifies the Small, Medium, and Large difficulty ordering for tabular tasks when they involve different datasets and domains?

3 comments

Thank you for recognizing the importance of the gaps we are addressing, the value of our benchmark and the extensiveness pf pir empirical evaluation. We address in the following your remaining concerns.

R3W1.1, R3Q1.1

In Task 4 (Contingency Analysis), the ground truth is determined by the largest absolute change in power flow

In the main paper, we defined Task 4 problem as "the model predicts which line overloads most"

In the appendix, we defined Task 4 problem as: "the model must identify the transmission line most likely to overload (experience the highest flow increase)"

We confirm that the formulation in the main paper is the correct one. Task 4 indeed consists at predicting which line is overloading (i.e., over line power boundaries), and not the one with highest increase of load.

We also confirm that all the results reported are correct both in the main paper and in appendix and reflect the proper formulation in the main paper.

We will update the appendix to correctly read: "the model must identify the transmission line most likely to overload (experience a power flow loading over the line flow boundaries). The standard line boundaries are defined in the problem as over 1.05 PU (105% nominal loads)."

R3W1.2, R3Q1.2

Task 10 (Constraint Expression Recovery) evaluates predictions against a specific reference expression

Thank you for raising this point. The objective of Task 10 was to check if the LLMs can recover one of the domain constraints established by the creators of the Tabularbench benchmark. However there was a confusion in the definition of the task in Appendix A.13.

Therefore we are now reporting for Task T10 two metrics:

  • Exact reconstruction: Whether the LLM recovered one of the human curated tasks, or any equivalent expression using the constraint parser of TabularBench. This is the metric our paper is currently reporting.

  • Validity: Whether the LLM recovered a constraint that is valid on the whole unseen test set of TabularBench.

URL dataset:

ModelReconstruction (%)Validity (%)
Llama 8B1.117.7
Qwen 8B6.246.5
DeepSeek-Llama 8B0.00.0
DeepSeek-Qwen 8B21.869.4
GPT-5.4-nano10.243.3
GPT-5.4-nano + Think6.524.9
Qwen 32B4.115.6
QwQ 32B20.037.3
GPT-5.4-mini13.933.6
GPT-5.4-mini + Think4.827.8
GPT-5.4 + Think14.669.8

LCLD dataset:

ModelReconstruction (%)Validity (%)
Llama 8B4.321.9
Qwen 8B3.558.3
DeepSeek-Llama 8B0.00.0
DeepSeek-Qwen 8B49.966.9
GPT-5.4-nano14.059.0
GPT-5.4-nano + Think54.176.5
Qwen 32B72.774.7
QwQ 32B87.392.7
GPT-5.4-mini47.878.7
GPT-5.4-mini + Think41.277.4
GPT-5.4 + Think57.587.4

CTU dataset:

ModelReconstruction (%)Validity (%)
Llama 8B21.250.8
Qwen 8B48.892.6
DeepSeek-Llama 8B0.00.0
DeepSeek-Qwen 8B42.961.8
GPT-5.4-nano61.478.4
GPT-5.4-nano + Think26.751.7
Qwen 32B47.368.6
QwQ 32B70.075.0
GPT-5.4-mini28.653.9
GPT-5.4-mini + Think28.453.1
GPT-5.4 + Think76.883.6

The validity results are much higher than the exact reconstruction, given that the LLM can return simple constraints to pass this metric.

We analyzed already this phenomonon in Appendix B.1. Our analyses showed for URL three recurring patterns: 1) latching onto a column that happens to be constant in the sample (nb_colon == 1), 2) restating a variable’s obvious domain or boundaries (ip * (ip-1) == 0), and 3) tightening a bound with slack observed in the sample (length_url >= length_hostname + 7 in place of length_hostname <= length_url).

On the other hand, if an LLM achieves 100% exact constraint reconstructionn it may hint to LLM memorization from the TabularBench paper and their repository. Indeed, they have disclosed the exact formulation of each expert-designed constraint.

Thank you again for raising this discussion. The current paper only reports the exact reconstruction metric. If these additional results adress your concerns, we can revise the manuscript accordingly and update the formulation in appendix A13 and the results in Table 17 in appendix with both metrics.

R3W1.3, R3Q2

Task 6 (Optimal Power Flow) primarily reports the proportion of individual constraints satisfied, rather than the percentage of complete solutions satisfying all constraints simultaneously

Currently no LLM manages to provide a single sample satisfying all constraints, accross all grid sizes. This makes the percentage of complete solutions satisfying all constraints simultaneously exactly 0% for all models and sizes.

We will include this 0% column in the revision.

R3W2.1, R3Q3

it is unclear how the observed performance compares with human experts, particularly on complex optimization tasks that ordinarily require computational tools

All the grid tasks of our benchmarks are real, and routinely run by grid operators using traditional computations tools for large grids. Tasks 1-5 can all be run without optimization tools.

  • Task 1 (path finding) & 2 (connectivity): Operators constantly need to know how power can physically reach a given area when lines are on outage (planned maintenance or faults). Checking whether parts of the network are energized or islanded is a continuous background function of a grid operator. After a trip, the system must detect islands (and flag dead buses) before any power flow can be solved. With a single-line diagram, an operator can trace a path or confirm connectivity in seconds, even with 1-2 lines removed for all the grids of our study (up to 118 bus). Note this is exactly why our paper’s small models failing here is notable: it is trivial for a human.

  • Task 3 (DC power flow): We restricted the evaluation to a DC linear flow. The DC model reduces to a linear system. For 14 buses you remove the slack bus and solve a 13x13 system, B'θ = P, then compute each line flow as (θi − θj)/x. With a calculator and a spreadsheet, it is feasible, but can take hours for manual calculation.

  • Task 4 (N-1 contingency): Power grid operators generate PTDF tables it becomes a few multiplications per line, so an operator could easily solve Task 4 for one contingency. PTDF (Power Transfer Distribution Factors) and LODF tables are precomputed sensitivity matrices derived from the DC power flow model (Task 3).

  • Task 5 (grid control). This is rule-based: read the frequency and voltage, match them to a playbook entry, and pick the action. It needs judgment and procedure knowledge, not computation, and it is precisely what operators are trained on.

  • Task 6 (OPF). The AC problem is non-linear and cannot be exactly solved without an optimlization algorithm. A human can only approximate it, for example by merit-order dispatch (cheapest generator first) and then checking limits, which gives a feasible but suboptimal answer.

We summarize below which tasks can be done by human operators:

TaskManualRealistic method
1 Path findingYes, easyTrace on the single-line diagram
2 ConnectivityYes, easyTrace on the single-line diagram
3 DC power flowYes, slowlySpreadsheet and calculator
4 ContingencyYes, slowlyPTDF/LODF tables
5 Power grid controlYes, easyPlaybook or operator judgment
6 OPFNoMerit-order approximation only, or optimization solvers

R3W2.2

some results may be influenced by output formatting and parsing rather than reasoning capabilities alone

For every task, the output formatting is specified and the LLM should respect it.

Task 5 for instance requires the answer to be formatted as [[A]]. A quick experiment asking for a JSON response instead ({"answer": "A"}, also a common format for LLMs) shows that this required formatting doesn't influence the scores:

ModelTask 5Task 5 - different formatting
Qwen3 8B71.4%70.8%
Qwen2.5 32B83.0%81.3%

R3W3.1, R3Q4

While the Small, Medium, and Large variants of graph-based tasks correspond to increasing network sizes (14, 30, and 118 buses), the corresponding variants of tabular tasks use entirely different datasets and application domains.

The Small, Medium, and Large benchmarks in the tabular setting reflect the classification and raking of these datasets in the original TabularBench benchmark.

In their original paper, they proposed datasets with increasing difficulty, based on the number and complexity of constraints.

URL (our S variant) is the simplest use-case with only 7 linear feature relationship constraints and 7 boolean single-feature constraints. Next is LCLD (our M variant) that has 19 single-feature immutability constraints, 3 linear and 7 non-linear feature relationship constraints. Finally, CTU, (our L variant) has 324 single-feature immutability constraints and 360 linear feature relationship constraints.

We agree with the reviewer that while the scale of constraints and relationship increases between the S, M, and L cases, the domain knowledge also chanes. Therefore, we propose to rephrase the tabular tasks as A, B, C rather than a classification by size Small, Medium, and Large.

Reviewer_i4jXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ORC-Bench evaluates LLMs on 30 graph and tabular tasks involving constrained reasoning and optimization. The authors find that valid outputs and satisfied variable bounds often coexist with violations of coupled constraints.

Critical strengths
  1. The focus on constraint satisfaction captures failures that answer accuracy and output validity miss.
  2. The graph and tabular tasks, together with the ablations, offer a useful diagnostic benchmark.
Critical weaknesses
  1. Task 10 asks for a valid constraint but scores recovery of a specific reference expression. The authors report that 93–95% of rejected constraints satisfy all displayed rows (B.1). Without held-out validation, low scores do not establish failure at constraint discovery.
  2. The OPF evaluation reports per-constraint satisfaction and MSE, leaving complete-solution feasibility and objective quality unverified. Coverage of reactive-power and line limits also remains unclear (A.7). This weakens the claim to evaluate end-to-end optimization.
Questions for authors
  1. How many rejected Task 10 constraints hold on independent rows, and would accepting these alternatives change the conclusions?
  2. Do solver outputs pass the same checker, and what are the models’ joint feasibility rates and objective gaps on feasible solutions?
  3. How do you count malformed outputs, and how much of the observed failure comes from parsing rather than incorrect answers?
4 comments
AuthorsOpenReview ↗

Clarifications on Task 10

Thank you for recognizing the usefulness of our benchmark and the relevance of our focus on constraint satisfaction. We have addressed your concerns and questions with additional results and we can update the manuscript accordingly if you do not have any further concerns or interrogations.

R2W1 / R2Q1: Task 10 asks for a valid constraint but scores recovery of a specific reference expression. How many rejected Task 10 constraints hold on independent rows, and would accepting these alternatives change the conclusions?

Thank you for raising this point. The objective of Task 10 was to check if the LLMs can recover one of the domain constraints established by the creators of the Tabularbench benchmark. However there was a confusion in the definition of the task in Appendix A.13.

Therefore we are now reporting for Task T10 two metrics:

  • Exact reconstruction: Whether the LLM recovered one of the human curated tasks, or any equivalent expression using the constraint parser of TabularBench. This is the metric our paper is currently reporting.

  • Validity: Whether the LLM recovered a constraint that is valid on the whole unseen test set of TabularBench.

URL dataset:

ModelReconstruction (%)Validity (%)
Llama 8B1.117.7
Qwen 8B6.246.5
DeepSeek-Llama 8B0.00.0
DeepSeek-Qwen 8B21.869.4
GPT-5.4-nano10.243.3
GPT-5.4-nano + Think6.524.9
Qwen 32B4.115.6
QwQ 32B20.037.3
GPT-5.4-mini13.933.6
GPT-5.4-mini + Think4.827.8
GPT-5.4 + Think14.669.8

LCLD dataset:

ModelReconstruction (%)Validity (%)
Llama 8B4.321.9
Qwen 8B3.558.3
DeepSeek-Llama 8B0.00.0
DeepSeek-Qwen 8B49.966.9
GPT-5.4-nano14.059.0
GPT-5.4-nano + Think54.176.5
Qwen 32B72.774.7
QwQ 32B87.392.7
GPT-5.4-mini47.878.7
GPT-5.4-mini + Think41.277.4
GPT-5.4 + Think57.587.4

CTU dataset:

ModelReconstruction (%)Validity (%)
Llama 8B21.250.8
Qwen 8B48.892.6
DeepSeek-Llama 8B0.00.0
DeepSeek-Qwen 8B42.961.8
GPT-5.4-nano61.478.4
GPT-5.4-nano + Think26.751.7
Qwen 32B47.368.6
QwQ 32B70.075.0
GPT-5.4-mini28.653.9
GPT-5.4-mini + Think28.453.1
GPT-5.4 + Think76.883.6

The validity results are much higher than the exact reconstruction, given that the LLM can return simple constraints to pass this metric.

We analyzed already this phenomonon in Appendix B.1. Our analyses showed for URL three recurring patterns: 1) latching onto a column that happens to be constant in the sample (nb_colon == 1), 2) restating a variable’s obvious domain or boundaries (ip * (ip-1) == 0), and 3) tightening a bound with slack observed in the sample (length_url >= length_hostname + 7 in place of length_hostname <= length_url).

On the other hand, achieving 100% exact constraint reconstruction can also be used as a sanity check to identify LLM memorization from the TabularBench paper and repository that disclosed the exact formulation of constraints.

Thank you again for raising this discussion. The current paper only reports the exact reconstruction metric. If these additional results adress your concerns, we can revise the manuscript accordingly and update the formulation in appendix A13 and the results in Table 17 in appendix with both metrics.

AuthorsOpenReview ↗

LLM’s joint feasibility rates and objective gaps on feasible solutions

R2W2.1 / R2Q2: The OPF evaluation reports per-constraint satisfaction and MSE, leaving complete-solution feasibility and objective quality unverified.

The solver outputs are included in the OPFDataset, used for this benchmark. All outputs from these datasets fully satisfy constraints on this checker.

Regarding joint feasibility, no LLM manages to provide a single sample satisfying all constraints simultaneously, across all grid sizes: the joint feasibility rate is 0% for all models and cases. This is consistent with the power flow constraint satisfaction (PF CTR), which never exceeds 14.4%.

Do solver outputs pass the same checker, and what are the models’ joint feasibility rates and objective gaps on feasible solutions?

Since no LLM solution is jointly feasible, objective gaps on feasible solutions cannot be computed. We therefore report, for every valid output, the absolute gap (Δ) between the generation cost of the predicted solution and the optimal cost from the solver, together with the PF CTR. We include the GNN baseline (Appendix B.4.6) as a reference predictor; for the GNN, the PF CTR column reports its overall constraint satisfaction (Table 15).

All numbers are expressed in $/h:

Case 14

ModelMean absolute Δ ± SEMin / max absolute ΔPF CTR (%) ↑
GNN6.84 ± 0.160.00 / 24.754.55 (overall)
Llama 8B130.99 ± 3.000.16 / 5361.0 ± 0.1
Qwen 8B113.94 ± 2.630.16 / 4642.8 ± 0.2
DeepSeek-Llama 8B134.99 ± 5.950.08 / 5283.9 ± 0.4
DeepSeek-Qwen 8B131.87 ± 5.230.00 / 2,2327.5 ± 0.3
Qwen 32B102.78 ± 2.600.00 / 4941.2 ± 0.1
QwQ 32B107.08 ± 2.450.08 / 3810.4 ± 0.1
GPT-5.4-nano106.88 ± 2.430.03 / 4147.6 ± 0.3
GPT-5.4-nano + reasoning94.01 ± 2.340.02 / 3844.3 ± 0.2
GPT-5.4-mini105.81 ± 2.480.24 / 4333.9 ± 0.2
GPT-5.4-mini + reasoning39.95 ± 2.030.04 / 1,4654.6 ± 0.2
GPT-5.4 + reasoning71.25 ± 2.890.03 / 4665.3 ± 0.2

Case 30

ModelMean absolute Δ ± SEMin / max absolute ΔPF CTR (%) ↑
GNN107.11 ± 2.500.35 / 40148.48 (overall)
Llama 8B545.25 ± 12.890.79 / 2,0020.2 ± 0.0
Qwen 8B526.43 ± 12.900.23 / 1,9200.2 ± 0.0
DeepSeek-Llama 8B546.06 ± 29.800.67 / 1,53610.6 ± 0.8
DeepSeek-Qwen 8B544.49 ± 13.931.30 / 1,93511.7 ± 0.4
Qwen 32B509.58 ± 12.860.14 / 2,1000.1 ± 0.0
QwQ 32B507.65 ± 11.730.29 / 1,7240.2 ± 0.0
GPT-5.4-nano545.65 ± 12.790.17 / 1,9803.2 ± 0.2
GPT-5.4-nano + reasoning483.95 ± 10.971.32 / 1,92714.4 ± 0.4
GPT-5.4-mini562.33 ± 13.610.04 / 2,4264.6 ± 0.2
GPT-5.4-mini + reasoning377.00 ± 11.260.18 / 3,11210.4 ± 0.4
GPT-5.4 + reasoning575.60 ± 14.592.94 / 2,15211.8 ± 0.3

Case 118

ModelMean absolute Δ ± SEMin / max absolute ΔPF CTR (%) ↑
GNN1,672.92 ± 38.153.56 / 7,47247.24 (overall)
DeepSeek-Qwen 8B2,394.69 ± 161.044.57 / 8,8932.8 ± 0.3
Qwen 32B2,674.55 ± 186.7723.20 / 24,0040.0 ± 0.0
QwQ 32B2,384.94 ± 91.590.53 / 28,8210.0 ± 0.0
GPT-5.4-nano2,462.64 ± 166.4030.75 / 153,3443.8 ± 0.1
GPT-5.4-nano + reasoning2,370.76 ± 57.892.03 / 25,2567.8 ± 0.0
GPT-5.4-mini2,381.10 ± 52.283.05 / 9,5184.1 ± 0.1
GPT-5.4-mini + reasoning2,561.64 ± 55.7417.82 / 8,8807.6 ± 0.0
GPT-5.4 + reasoning2,487.08 ± 161.891.12 / 49,7488.0 ± 0.1

All LLMs remain far from the optimal cost, well behind the GNN baseline. Reasoning reduces the gap on the smaller grids but not on the largest one, and a lower cost does not come with better feasibility.

We can add these cost tables to Appendix B.4, and update the Task 6 discussion accordingly.

AuthorsOpenReview ↗

Clarifications on coverage of reactive-power and line limits, and on parsing failures

R2W2.2: Coverage of reactive-power and line limits also remains unclear (A.7).

Table 7, containing the full detailed results (appendix), show that bound constraints (reactive power and line limits) are almost always respected by all LLMs (average constraint satisfaction of 99.8%, the worst LLM performing at 97.9%). We thus focus on the most difficult constraint, the satisfaction of powerflow equations, which is much harder for LLMs to satisfy.

R2Q3; How do you count malformed outputs, and how much of the observed failure comes from parsing rather than incorrect answers?

A specific pattern is asked to the LLM for the output (more details in appendix). The first matching pattern is extracted and parsed to compare it to the ground truth. If no matching pattern is found, this is considered as a malformed output. We will ensure this is clearly explained in the paper.

Dear reviewer,

We hope that our additional, fine-grained metrics for Task 10 and the clarifications regarding OPF solution feasibility has addressed your concerns. We have updated the paper accordingly. We remain available for any additional clarification.

Reviewer_knrSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ORC-Bench, a benchmark for evaluating LLMs' reasoning and optimization capabilities under complex constraints. It includes 30 tasks across ten categories and three difficulty levels, covering power system analysis, optimization, tabular data, and constraint discovery. The authors evaluate different LLMs using accuracy, prediction error, and constraint satisfaction metrics. The results highlight the limitations of current LLMs in handling complex physical constraints, particularly in OPF tasks. While the benchmark provides a useful evaluation resource, the validity of some task designs, the reliability of the evaluation methods, and its unique contributions compared with existing benchmarks require further clarification.

Critical strengths
  1. The paper addresses an important problem by evaluating LLMs' reasoning and optimization capabilities under complex constraints. It brings together tasks such as grid topology analysis, power flow calculation, contingency analysis, and constraint discovery into a unified benchmark, offering broad coverage and practical relevance.

  2. The paper uses programmatic data generation to create evaluation tasks of different sizes and difficulty levels, and makes the dataset publicly available. This provides a reusable resource for future research and makes it easier for other researchers to compare and extend the evaluation.

Critical weaknesses
  1. Some power system tasks do not fully align with the capabilities they aim to evaluate. Specifically, (1) Task 4 identifies the line with the largest power flow change as the most overloaded line, although these are not equivalent; (2) Task 5 maps predefined operating states to control actions, making it closer to a classification task than actual power grid control; and (3) Task 6 mainly evaluates prediction errors and constraint satisfaction rates, without adequately assessing the optimization objective or optimality gap. These issues may limit the practical relevance of the evaluation results.

  2. According to the Appendix and the publicly available Dataset Card, Tasks 4 and 5 are evaluated by checking whether the correct answer appears in the model output. Could a model receive a correct score by listing multiple candidate answers? The authors should clarify the scoring procedure and verify whether such strategies could artificially inflate the reported accuracy.

Questions for authors
  1. Several LLM benchmarks for power systems already exist, and some have reported findings similar to those presented in this paper. However, the differences between ORC-Bench and existing benchmarks are not sufficiently discussed. The authors should clarify the contributions of ORC-Bench and explain why a new benchmark is needed.

  2. Although the authors use random perturbations and scenario generation to reduce the risk of memorization, the IEEE 14/30/118-bus systems are widely available standard test cases. It remains unclear whether the models rely on memorized network structures or fixed generation patterns. Additional tests with unseen network topologies, bus renumbering, and new constraint types would help assess their reasoning and generalization capabilities.

  3. Power system applications rely heavily on mathematical and physical models/tools, e.g., power flow solvers, optimization solvers, and verification tools. Although the authors provide preliminary tool-assisted experiments, their scope remains limited. Could the authors further compare LLMs with tool-augmented and agent-based approaches to distinguish limitations in numerical computation from the ability to solve engineering problems using external tools?

  4. The evaluated models do not fully cover representative frontier LLMs, for example, Claude and Gemini. I suggest the authors evaluate stronger models to determine whether parts of ORC-Bench are approaching saturation and whether the benchmark can still distinguish between advanced models.

10 comments

Thank you for recognizing the relevance of our problem, and the value of our benchmark. We believe our manuscript led to a few misunderstandings that we clarify in the following. We have also included additional evaluations requested and we will revise the submitted pdf if our answers adress your concerns.

R1W1

Misalignment of Task 4 description in Appendix with the main paper

In the main paper, we defined Task 4 problem as "the model predicts which line overloads most"

In the appendix, we defined Task 4 problem as: "the model must identify the transmission line most likely to overload (experience the highest flow increase)"

We confirm that the formulation in the main paper is the correct one. Task 4 indeed consists at predicting which line is overloading (i.e., over line power boundaries), and not the one with highest increase of load.

We also confirm that all the results reported are correct both in the main paper and in appendix and reflect the formulation in the main paper.

We will update the appendix to read: "the model must identify the transmission line most likely to overload (experience a power flow loading over the line flow boundaries). The standard line boundaries are defined in the problem as over 1.05 PU (105% nominal loads)."

R1W2

Task 5 is closer to a classification task than actual power grid control

We would like first to argue than a task type (classification/regression/open-ended question) does not contradict the purpose of the task (here power grid control). Those are longitudinal axies. Therefore a task can be a power grid control defined as a multiple choice classification.

Task 5 evaluates the selection among 4 candidate actions that vary across queries; these candidate actions are distinct for every query in the dataset. We will clarify this in the task definition.

Selecting a corrective action from the observed grid state is a component of grid control, naturally formulated as classification over discrete actions, assessed here under predefined operating rules.

R1W3

Task 6 mainly evaluates prediction errors and constraint satisfaction rates, without adequately assessing the optimization objective or optimality gap.

Thank you for suggesting these additional metrics. We restricted the evaluation to traditional machine learning metrics, and we have run for the rebuttal additional evaluation to adequately address your request.

First, we provide the cost of the solutions predicted by the models (optimization objective, in $/h) and their comparison with the reference solver across all 3 grid sizes, reporting mean absolute errors for every setup, standard errors and ranges.

However, cost comparison alone between predicted and solver's solutions does not establish optimization quality: LLMs can predict lower-cost solutions by violating constraints, whereas the solver always returns feasible solutions. These results must therefore be interpreted alongside constraint satisfaction; a meaningful optimality comparison requires feasibility first.

The optimal solution by the LLM should approach the ground truth cost. Our results below demonstrate exactly this claim: Only the GNN baselines find solutions with cost close to the optimal ground truth solutions.

Case 14

ModelMAE ± stderrMin / max absolute Δ
GNN6.84 ± 0.160.00 / 24.7
Llama 8B130.99 ± 3.000.16 / 536
Qwen 8B113.94 ± 2.630.16 / 464
DS-Llama 8B134.99 ± 5.950.08 / 528
DS-Qwen 8B131.87 ± 5.230.00 / 2,232
Qwen 32B102.78 ± 2.600.00 / 494
QwQ 32B107.08 ± 2.450.08 / 381
GPT-5.4-nano106.88 ± 2.430.03 / 414
GPT-5.4-nano + reasoning94.01 ± 2.340.02 / 384
GPT-5.4-mini105.81 ± 2.480.24 / 433
GPT-5.4-mini + reasoning39.95 ± 2.030.04 / 1,465
GPT-5.4 + reasoning71.25 ± 2.890.03 / 466

Case 30

ModelMAE ± stderrMin / max absolute Δ
GNN107.11 ± 2.500.35 / 401
Llama 8B545.25 ± 12.890.79 / 2,002
Qwen 8B526.43 ± 12.900.23 / 1,920
DS-Llama 8B546.06 ± 29.800.67 / 1,536
DS-Qwen 8B544.49 ± 13.931.30 / 1,935
Qwen 32B509.58 ± 12.860.14 / 2,100
QwQ 32B507.65 ± 11.730.29 / 1,724
GPT-5.4-nano545.65 ± 12.790.17 / 1,980
GPT-5.4-nano + reasoning483.95 ± 10.971.32 / 1,927
GPT-5.4-mini562.33 ± 13.610.04 / 2,426
GPT-5.4-mini + reasoning377.00 ± 11.260.18 / 3,112
GPT-5.4 + reasoning575.60 ± 14.592.94 / 2,152

Case 118

ModelMAE ± stderrMin/max absolute Δ
GNN1,672.92 ± 38.153.56/7,472
DS-Qwen8B2,394.69 ± 161.044.57/8,893
Qwen32B2,674.55 ± 186.7723.20/24,004
QwQ32B2,384.94 ± 91.590.53/28,821
GPT-5.4-nano2,462.64 ± 166.4030.75/153,344
GPT-5.4-nano + reasoning2,370.76 ± 57.892.03/25,256
GPT-5.4-mini2,381.10 ± 52.283.05/9,518
GPT-5.4-mini + reasoning2,561.64 ± 55.7417.82/8,880
GPT-5.4 + reasoning2,487.08 ± 161.891.12/49,748

These results reinforce our finding that low prediction error or apparent cost savings cannot substitute for physical feasibility: the GNN baseline achieves the lowest cost MAE across all three grids, while LLM predictions remain substantially farther from the reference costs.

R1W4

For Task 4, could a model receive a correct score by listing multiple candidate answers?

Thank you for highlighting this potential breach. Our manual inspection found no instances of models exploiting this behavior in the reported experiments. We have now added safeguards against multiple answers: Only the first line returned by the LLM is considered. An LLM returning multiple answers that include the correct solution later in the answer will be considered wrong.

If you consider this mitigation adequate, we will update the the scoring procedure in the appendix accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers for their questions and feedbacks. We have addressed all the concerns and revised the PDF. The revision includes:

Main text

  • tabular variants renamed A/B/C
  • Task 4 definition clarified, with only the first answer scored;
  • clearer definition of Task 5 (candidate actions differ per scenario)
  • handling of malformed outputs made explicit

Appendix

  • OPF joint feasibility and cost gap (A.7);
  • A/B/C constraint counts (A.8-A.10);
  • Task 10 Exact reconstruction and Validity metrics (A.13)
  • B.2.3: Task 4 sub-tasks (flow increase vs. actual overloads)
  • B.2.4: unseen private 485-bus grid and bus renumbering ablation
  • B.2.5: output-format ablation
  • B.2.6: comparison with human operators
  • B.4: OPF joint feasibility (0% for all models) and cost gap vs. the solver and the GNN
  • B.4.5: code execution and agentic evaluation.
  • B.5: comparison with Claude models
  • Table 27: Task 10 Validity

These ablations confirmed the conclusions of the paper, which remain unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.