acceptodds
Under review as a conference paper at ICLR 2027

FULL-FORMULATION DC-OPF SYNTHESIS: A probe into how LLMs fail in physical modeling

Abstract

We introduce full-formulation direct-current optimal power flow (DC-OPF) synthesis as a probe of how reliably large language models (LLMs) construct mathematical models of physical systems. The task requires an LLM to construct a complete executable formulation from a natural-language dispatch instruction and an explicit modeling contract, without a supplied OPF backbone. We construct a benchmark of 118 instructions over three PGLib systems and develop a certificate-based evaluator for instance-level formulation equivalence. Its bidirectional violation linear programs detect discrepancies in hard feasible regions even when candidate and reference optimal values agree. We precisely specify the mathematical objects assessed by the evaluator and the scope of certification. Across nine LLMs, 244 of 1,062 outputs fail the certificate checks. A source-level fidelity review of all 244 failed outputs identifies substantive modeling errors in every evaluated model. These findings motivate a different allocation of modeling responsibilities: the LLM interprets the operational meaning of a natural-language requirement, while deterministic software binds it to concrete network entities and instance-dependent data and constructs the complete DC-OPF. A protocolized workflow implementing this allocation raises aggregate certificate passes from 818/1,062 to 1,054/1,062.

Primary area:
datasets and benchmarks
Keywords:
Large language model evaluation, Physical modeling, Optimization modeling, Instance-level formulation equivalence, Certificate-based evaluation, DC optimal power flow

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yuCTOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores using large language models (LLMs) to construct complete, executable mathematical models of physical systems. Specifically, in this work, full-formulation DC-OPF synthesis is used, aiming to get an LLM to construct a complete executable formulation from a natural-language dispatch instruction and an explicit modeling contract, without a supplied OPF backbone. In the paper, the authors formalize this problem, develop a certificate-based evaluator, and test several LLMs. The results indicate that there are substantive physical-modeling errors, and propose a different pipeline, where the modeling responsibilities are such that LLMs are used to interpret natural language requirements, while deterministic software does the actual construction of the complete DC-OPF. The proposed pipeline improved results (fewer errors).

Critical strengths
  • The proposed pipeline results in fewer errors, thus is a viable and useful result and insight for the application of LLMs for this DC-OPF problem.
  • Evaluation includes human expertise for validating the input to the LLM, and also the resulting outputs.
  • The evaluation also considered different strategies and proposed pipeline to demonstrate the improved performances and analyse the amount of inference tokens (costs).
  • The overall high-level story and objective of the paper was clearly presented and, therefore, easy to understand.
Critical weaknesses
  • The paper uses LLMs without specific tuning for the given problem, and the results of 818/1062 indicate that 77% of the time the LLM produced correct outputs. What error rate is acceptable in practice, and how do the LLMs compare with other methods commonly used for this modeling?
  • The proposed pipeline is hand-engineered and it seems to me that it is expected that it works better. Giving the LLM more opportunity to improve the way it generates the outputs for the full-formulation by allowing tool calls and execution of the code for validation, it would also perhaps lead to better performance. Thus, adding, for example, a fine-tuned LLM for this problem and an LLM with tool calls and ability to execute and test code, for comparison with the proposed pipeline and baseline would be beneficial.
  • Motivation to use LLMs in this way (for the full-formulation baseline) is unclear. Why do we want to ask an LLM to write a whole DC-OPF from scratch in practice, when there are existing tools for that?
  • The evaluation is strictly binary (pass/fail), so an output with one minor error scores the same as one that is entirely wrong. Partial-credit or severity information would help judge how close the models are to successful modeling.
  • Evaluation seems to include single runs of LLMs (Table 1). Multiple runs should be executed with statistics on the outcomes. Also, initial experiments are conducted with 9 models, while later analysis and comparison is done with only 1-2 models, which decreases the generalizability of the findings.
  • While the overall high-level understanding of the objective of the paper is clear, it was hard to understand the details of the work. Many abbreviations are not defined (e.g., JDIMP, JDEF). Moreover, adding more explanation on the specific cases used (case30, case118, case300) could lead to better understanding of the work (to those that are not familiar with the PGLib library). Moreover, a lot of the work is shown only in the very long appendices. Figure 1 uses a lot of space for little information. Some figure text in Figure 5 is clipped and unreadable.
  • While this is an applied LLM paper, it is likely more valuable to the DC-OPF community than to ICLR. The paper offers practical information for DC-OPF researchers on how to better leverage existing LLMs for their domain, while insights about LLMs are fairly limited.
Questions for authors

Please address the stated weaknesses.

5 comments

Response (1/2)

We thank the reviewer for the constructive comments.

1. Research objective and relevance (W1, W3, W7)

Physical-modeling capability provides evidence of whether general-purpose LLMs can organize knowledge of the physical world into reliable foundations for reasoning and decision-making. We use DC-OPF as a mathematically verifiable probe of this capability. DC-OPF combines substantive physical-modeling demands with mathematically tractable verification, allowing us to distinguish incorrect formulations from merely different representations.

Our full-formulation task requires constructing complete executable formulations under an explicit modeling contract. It directly tests whether LLMs can construct the power-flow equations, nodal balances, unit conversions, and standard constraints that backbone-based tasks already provide. Across nine LLMs, we identify 90 first-attributed physical-relation failures, including 38 missing MW/per-unit conversions, 21 incorrect impedance-base conversions, and 18 branch-direction/incidence-sign errors. Separately, 49 failures concern entity binding and scope (§6.2, App. L.1). These findings expose concrete difficulties in both physical-relation construction and binding requirements to network entities.

Together, formulation checks and source-level error analysis measure the reliability of LLM physical-model construction and identify where it fails. The observed success rate measures formulation reliability; individual models intended for practical use require correctness checks rather than acceptance based on an aggregate success rate. In safety-critical applications such as power-system operation, even a single error in representing physical constraints may lead to unsafe operating decisions.

2. Progressive interventions reveal failure mechanisms (W2)

We agree that deterministic construction is expected to improve reliability. Our contribution is the empirical investigation of which modeling operations cause errors and which errors persist when those operations are handled deterministically.

We use “construction errors” to denote the combined count of failures first attributed to physical relations, constraint semantics, or program implementation. Each failure is counted once.

We progressively introduce local physical primitives, complete constraint construction, and deterministic entity binding (§6.4, App. H.3). Across DeepSeek V4 Flash and GPT-5.6 Luna, construction errors decrease 65 → 25 → 1 across the first three conditions, while entity-binding/scope errors remain 17, 18, and 22.

These results reveal two distinct difficulties: correct local physical coefficients do not guarantee correct assembly of flow and balance equations; removing equation-construction errors does not eliminate incorrect entity grounding. The proposed allocation is informed by these observed failure mechanisms rather than merely by the expectation that a deterministic backend performs better.

3. New execution-feedback baseline (W2)

We additionally evaluate DeepSeek V4 Flash with execution feedback and two rounds of revision. This condition achieves 106/118 certificate passes, compared with 101/118 for single-call synthesis, 113/118 for reflection, and 112/118 for two builders plus an integrator. We will include this experiment in the revised paper.

The 12 remaining failures include six incorrect state updates, four missed rejections, one Python call error, and one nodal-balance sign reversal. In case118_A_006, the generated model matches the reference optimal objective, yet a violation-LP witness identifies a 102 MW nodal-balance discrepancy.

Execution and solver feedback can identify executable-code failures without establishing the correctness of the underlying physical formulation. This provides additional evidence for the distinction between successful execution and faithful physical modeling.

Our evaluation deliberately examines general-purpose LLMs without task-specific tuning. Fine-tuning investigates specialized adaptation, whereas these experiments examine existing modeling capabilities and the effects of inference-time intervention.

Response (2/2)

4. Correctness certification and error diagnosis (W4)

The certificate outcome is binary because certification requires all applicable correctness checks to pass. For feasible tasks, transformed states, objectives and soft epigraphs, and complete hard feasible regions must agree with the reference within the specified numerical tolerances to establish instance-level formulation equivalence. The pass rate reports the proportion of candidates satisfying the required checks.

Our analysis goes beyond this binary outcome to characterize the errors and their mathematical consequences. We review all 244 full-synthesis certificate failures and classify each by its first confirmed error: 90 in physical relations, 49 in entity binding and scope, 36 in constraint semantics, and 69 in program implementation. Bidirectional violation LPs additionally provide concrete witnesses of feasible-region discrepancies. Among 26 formulations rejected by feasible-region checks, 11 have matching candidate-declared, externally reconstructed, and reference optimal values (§4, App. L.2).

5. Repeated runs and presentation (W5, W6)

Appendix H.5 already reports three independent complete runs for GPT-5.6 Luna and Gemini-3.6 Flash under both evaluated conditions. Full-synthesis certificate passes are 53/52/53 for Luna and 78/83/85 for Gemini, directly characterizing run-to-run variability. The nine-model evaluation establishes the breadth of observed failures, while the targeted experiments with fixed models examine specific error mechanisms and interventions.

We will define JDIMP/JDEF at first use, better explain the PGLib network cases, reduce Figure 1, and improve the readability of Figure 5.

Thank you for the detailed response and the additional experiment.

  1. In your response you clarify the research objective of this work, but together with how the paper currently reads, this framing makes the objective less clear to me. Specifically, it is not clear whether the paper is primarily an applied study of LLMs for DC-OPF with specific DC-OPF findings, a new method (the proposed pipeline) for DC-OPF modeling, or a benchmark for evaluating LLMs. The paper currently has elements of all three, and these would be assessed differently. If the benchmark is a main contribution, the paper would benefit from guidance on how others should use it: a standardized evaluation protocol, how the benchmark can be extended or used, and more explanation for ML researchers without power-systems expertise. It would also need a clearer comparison with existing related benchmarks mentioned in the related work, such as ProOPF and Text2Opt-Bench: how similar the tasks are, which insights are new and which confirm prior findings, and what the benefit is of adding another closely related problem. If the main contribution is the applied study or the pipeline, could you clarify which insights transfer to the broader ML community beyond DC-OPF?
  2. You frame the main objective of the paper as a test of LLMs’ physical-modeling capability ("organize knowledge of the physical world into reliable foundations for reasoning and decision-making"). Since the modeling contract provides the model variant, conventions, and units, the task seems to test faithful implementation of a specified physical model rather than the choice of how to model a physical system (which laws, approximations, and assumptions apply). Can you clarify this difference and how it affects the claims in the title and conclusions?
  3. The new experiment is a useful addition. Could you report its number of calls and inference tokens, as in Table 2? Could you also classify the 12 remaining failures using the same four categories as Table 1 (physical relations, entity binding and scope, constraint semantics, program implementation), for consistency and to better understand which error types the feedback helps with? For DeepSeek V4 Flash, feedback reduces failures from 17 to 12, which is a modest difference from a single run, and below the reflection strategy (113/118). Since DeepSeek V4 Flash is among the stronger models in Table 1, there is limited room for improvement. If feasible within the discussion period, running the baseline for GPT-5.6-Luna, which had 65 failures, would show more clearly whether execution feedback meaningfully helps.

As a suggestion for the revision, I would recommend adding examples of the prompts and outputs (prompt instruction, and a successful and failing output). This would add additional information and clarity on the tasks for LLMs, especially for readers outside the power-systems field.

Points 1 and 2 are more important for my assessment, and at the moment I still consider the paper borderline.

Reviewer_1597OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether LLMs can construct complete, executable mathematical models of physical systems from natural-language instructions and an explicit modeling contract, without being given an OPF backbone. The authors introduce a DC-OPF benchmark with 118 instructions derived from three PGLib cases, together with a certificate-based evaluator that checks instance-level formulation equivalence beyond agreement in optimal objective values. They evaluate nine LLMs, identify 244 failures, and analyze them across four categories: physical relations, entity binding and scope, constraint semantics, and program implementation.

Critical strengths

The paper extends prior LLM-based OPF modeling benchmarks in two relevant ways: it evaluates full-formulation construction without a supplied OPF backbone, and it assesses formulation equivalence beyond matching optimal values. In particular, the certificate-based evaluation checks whether the transformed state, objective and soft epigraphs, and complete hard feasible region are correctly modeled. This provides a more detailed assessment of model correctness than relying on optimal-value agreement alone.

Critical weaknesses
  1. The paper’s central task, full-formulation DC-OPF synthesis, is distinct from backbone-based OPF modeling, but the paper does not clearly establish what additional failure modes or insights this task reveals. It is also unclear whether removing the supplied backbone constitutes a substantial benchmark contribution or a relatively direct extension of prior tasks. The paper should distinguish the contribution of the new task from that of the more detailed evaluator.

  2. The paper emphasizes its proposed allocation of modeling responsibilities, but the reported failures attributed to this allocation appear only eight times across the nine models. The paper does not sufficiently explain why this aspect is central to the benchmark or how these cases support the emphasis placed on it.

Questions for authors
  1. Why does Table 1 sum the number of failures across all models? This reviewer thinks it is more important to know how each model performs on the 118 tasks, where it fails, and what causes those failures. As presented, the table may be confusing to readers. Could the authors clarify the rationale for this presentation?

  2. The 118 tasks seem relatively few for a dataset. Therefore, it would be helpful to include a table comparing this benchmark with similar existing datasets in terms of their similarities, differences, and scale. This reviewer saw Table 7 in the appendix, but a comparison with other similar benchmarks would be more useful.

  3. Table 1 reports only eight proposed-allocation failures across the nine models. Thus, this reviewer is wondering how, or why, the number of aggregate certificate passes increases from 818/1,062 under full-formulation synthesis to 1,054/1,062.

  4. This is a somewhat minor point, but this reviewer feels that Figure 1 is a bit overstated. This reviewer thinks that the main difference between prior OPF modeling tasks and the task proposed in this paper seems to be whether an OPF backbone is provided. This may be analogous to the difference between constructing a building from scratch and modifying or extending an existing building. If this is not the case, could the authors clarify what makes the proposed task substantially different?

1 comment

We thank the reviewer for their comments.

1. Why full-formulation DC-OPF synthesis? (W1, Q4)

We use DC-OPF as a nontrivial yet mathematically tractable probe of LLM reliability in constructing executable mathematical models of physical systems. Backbone-based OPF tasks, including ProOPF, meaningfully evaluate instruction interpretation and model modification. However, their supplied formulations already encode the underlying power-flow equations, unit conversions, and balance constraints. Successfully using or extending such a formulation therefore does not directly test whether the LLM can construct those physical relations itself.

To expose this capability, our baseline removes the supplied OPF backbone and domain-specific model-construction tools, while retaining an explicit modeling contract and generic Pyomo guidance. This design makes physical-relation construction directly testable: across nine LLMs, full synthesis yields 90 first-attributed physical-relation failures, including 38 missing MW/per-unit conversions, 21 incorrect impedance-base conversions, and 18 branch-direction/incidence-sign errors (§6.2, App. L.1). A prebuilt OPF backbone supplies these standard conversions and equations, so such vulnerabilities may remain unobserved when these modeling responsibilities are delegated to the backbone.

The task and evaluator are already distinguished as separate contributions (Contributions 1–2; §§3–4). The task exposes physical-model construction to assessment; the evaluator tests formulation equivalence beyond the optimal value. Of 26 candidates rejected by bidirectional feasible-region checks, 11 match the candidate-declared, externally recomputed, and reference optimal costs (§4, App. L.2). Fig. 1 illustrates the difference in formulation responsibilities—modifying an existing model versus constructing one from specifications. This distinction is consistent with the reviewer’s analogy.

2. Table 1 and the eight residual failures (W2, Q1, Q3)

The “eight failures” are the residual failures under the proposed allocation. Full synthesis has 244 failures, first attributed to physical relations (90), entity binding and scope (49), constraint semantics (36), and program implementation (69). The proposed allocation reduces the failure count to 8, a net gain of 236 passes (818/1,062 → 1,054/1,062). The paired accounting is in App. L.3 (Tables 37–39).

Table 1 reports each model's counts across all four failure categories; the aggregate summarizes the nine models × 118 tasks. We will make the two experimental-condition headings more prominent.

3. What do changing failure patterns reveal about responsibility allocation? (W2,Q3)

To understand this improvement, we examine how the remaining failures change as specific modeling responsibilities are progressively transferred to deterministic software (§6.4).

We use “construction errors” to denote the combined count of failures first attributed to physical relations, constraint semantics, or program implementation. Each failure is counted once.

Responsibility allocationCertificate passesConstruction errorsEntity binding and scope
Full synthesis154/2366517
Primitive helper193/2362518
Constraint helper213/236122
Proposed workflow235/236——

Results pool DeepSeek V4 Flash and GPT-5.6 Luna.

Across the two models, construction errors decrease from 65 under full synthesis to 25 with the primitive helper and 1 with the constraint helper. The primitive helper supplies correct local coefficients and directions, yet LLM-generated code still assembles some flow and nodal-balance equations incorrectly. Under the constraint helper, entity binding and scope account for 22 of the 23 remaining failures, while instance-dependent binding remains the responsibility of generated code.

The proposed workflow also transfers instance-dependent binding to deterministic software and achieves 235/236 certificate passes. These results support delegating both instance-dependent binding and complete DC-OPF construction to deterministic software, while retaining the LLM’s role in interpreting natural-language requirements.

4. Benchmark comparison (W1, Q2)

A brief benchmark comparison follows; we will expand it in the revision.

BenchmarkTest casesVariable scalePrimary TaskEvaluation Criterion
IndustryOR10010^0–10^2General industrial OR modelingOptimal-value matching
ProOPF-B12110^1–10^3Modify/extend an OPF backboneOptimal-value matching
Ours11810^1–10^3Full DC-OPF constructionInstance-level formulation equivalence
Reviewer_jLKvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLMs can construct complete DC-OPF models from natural-language instructions without being given a predefined OPF formulation. The authors introduce a benchmark of 118 instructions across three PGLib network instances and evaluate nine LLMs using a certificate-based evaluator. The evaluator checks the equivalence of the generated and reference formulations, rather than relying only on whether they produce the same optimal objective value. The authors find that 244 out of 1,062 generated formulations fail the certificate checks. They analyze these failures and propose a workflow in which the LLM interprets the instruction, while deterministic software handles entity binding and model construction. This reduces the number of certificate failures to 8 on the same benchmark.

Critical strengths
  1. The evaluation goes beyond comparing optimal objective values. Two optimization models may achieve the same optimal objective value while imposing different constraints and allowing different feasible operating regions. The proposed certificate-based evaluator addresses this issue by checking the equivalence of hard feasible regions. This is a useful contribution to the evaluation of LLM-generated optimization models.
  2. The paper provides a systematic analysis of modeling errors. The authors evaluate nine LLMs, categorize their modeling errors, and compare several alternative approaches with the proposed workflow. The results show that delegating well-defined modeling operations to deterministic software can substantially improve reliability on the evaluated benchmark. These experiments also help identify where errors arise and which parts of the task benefit most from deterministic handling.
Critical weaknesses
  1. The methodological novelty of the proposed workflow remains unclear. The proposed workflow is a reasonable engineering design, and the reported performance improvement is substantial. However, the workflow itself appears relatively straightforward. The paper does not sufficiently establish what methodological contribution it offers beyond combining an LLM with a domain-specific modeling backend. Although the proposed benchmark and certificate-based evaluator may offer more distinctive contributions, their advantages over existing optimization-modeling benchmarks and verification methods are not sufficiently demonstrated through direct comparisons.
  2. The evaluation scope is limited. The experiments use 118 instructions across three PGLib network instances and rely on a predefined modeling protocol and a library of deterministic components. Although the tasks cover various operational requirements, the results do not establish whether the proposed workflow generalizes to unseen modeling mechanisms, substantially different network instances, or other physical optimization problems. Evaluation on independent tasks or additional network instances would help clarify the extent to which the findings generalize.
Questions for authors
  1. How does the proposed certificate-based evaluator improve upon existing optimization-model verification methods? What important errors can it detect that existing benchmarks or evaluation approaches may miss?
  2. How were the 118 instructions used during the development of the proposed workflow? In particular, are the four in-context examples drawn from the evaluated task set?
2 comments

We thank the reviewer for recognizing our formulation-level evaluation and systematic error analysis.

1. Advantages of certificate-based evaluation (W1, Q1)

Our evaluator addresses a fundamental limitation of optimal-objective-based correctness assessment: two formulations can achieve identical optimal values while representing different feasible regions. ProOPF (Shen et al., ICML 2026), the closest OPF modeling benchmark, determines end-to-end correctness through optimal-value agreement.

Our experiments demonstrate the physical-modeling errors this criterion misses. Among 26 formulations rejected by our bidirectional feasible-region checks, 11 have matching candidate-declared, externally reconstructed, and reference optimal values. For case30_A_029, all three costs agree at 8,036.3056, yet violation LPs reveal nodal-balance discrepancies of 35.4584 MW and 17.6235 MW.

Compared with other verification methods, EquivaMap uses LLM-proposed variable mappings and verifies mapped optimal solutions; our evaluator deterministically aligns physical variables and checks entire hard feasible regions. ModelEquivBench uses Farkas containment certificates (dual to our violation LPs), but its exhaustive permutation coverage is stated only for n ≤ 7 without binary coordinates. Our public physical identities eliminate mapping search, enabling bidirectional checks on 32-, 137-, and 357-variable DC-OPF models.

2. Methodological contribution of responsibility allocation (W1)

While combining an LLM with a deterministic backend is not itself a novel architecture, our contribution is to empirically identify which modeling responsibilities should be delegated, and why. Motivated by our audit of 244 full-synthesis failures across nine LLMs, we progressively transfer modeling responsibilities to deterministic components.

We use “construction errors” to denote the combined count of failures first attributed to physical relations, constraint semantics, or program implementation. Each failure is counted once.

Responsibility allocationCertificate passesConstruction errorsEntity binding and scope
Full synthesis154/2366517
Primitive helper193/2362518
Constraint helper213/236122
Proposed workflow235/236——

Results pool DeepSeek V4 Flash and GPT-5.6 Luna.

The experiments reveal two distinct bottlenecks. First, the primitive helper supplies correct local physical coefficients and directions, yet LLMs still assemble incorrect global flow and nodal-balance equations. Second, the constraint helper constructs these equations deterministically, reducing construction errors from 65 to 1, but 22 entity-binding/scope errors remain because binding is still handled by generated code. The proposed workflow also delegates instance-dependent binding to deterministic software.

These results explain why correct physical primitives alone are insufficient and why deterministic construction must be complemented by entity grounding.

3. Evaluation scope and generalization (W2)

Appendix G evaluates nine independently authored requests on the much larger case1354_pegase network (1,354 buses, 260 generators, 1,991 branches). An isolated LLM generated requests using the declared DC-OPF scope and raw network facts, without access to our protocol, benchmark tasks, examples, implementation, or feedback.

Eight of nine requests were expressible under the unchanged protocol; all eight passed end-to-end certificate checks, including one correctly identified as infeasible. The ninth required updating stored angle bounds not exposed by the protocol; a separate check confirmed correct backend construction after that update. These results demonstrate successful modeling of independently authored request combinations on a substantially larger network, beyond the original 118 benchmark instructions.

4. Benchmark development and examples (Q2)

Preliminary full-formulation synthesis experiments on draft tasks revealed recurring modeling failures and informed the workflow design. The workflow and its four-shot prompt were developed before construction of the final 118-task benchmark, so the examples were not selected from the evaluated tasks. After the 118 instructions were frozen, no evaluation-driven changes were made to the workflow.

Furthermore, a controlled ablation with DeepSeek V4 Flash (App. H.4) shows that removing all four examples while keeping everything else fixed still yields 115/118 certificate passes, compared with 118/118 using four-shot prompting. This confirms that the workflow's high reliability does not depend on those examples.

I thank the authors for the clarifications. Q1 and Q2 are largely resolved. However, the two critical weaknesses remain only partially mitigated, and the paper still lacks direct systematic comparison with existing benchmarks/verification methods. I therefore maintain my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Clarifying the Research Focus

We thank the reviewers for their thoughtful comments and would like to clarify the research objective connecting our benchmark, evaluator, and workflow experiments. Our paper studies how reliably large language models construct mathematical models of physical systems. Here, constructing a model means translating a description of a physical system and its requirements, under stated physical assumptions, into a complete executable mathematical formulation. Physical-modeling capability provides evidence of whether general-purpose LLMs can organize knowledge of the physical world into reliable foundations for reasoning and decision-making. We investigate this capability in general-purpose LLMs using full-formulation DC-OPF synthesis as a probe.

Choosing such a probe involves a trade-off between the coverage of physical modeling problems and the accuracy of the resulting capability assessment. The importance of the latter is illustrated by Ansari et al. (2026), whose audit of existing physics benchmarks shows that grading errors and flawed benchmark materials can substantially distort apparent model performance. Their findings demonstrate that benchmark scores cannot be interpreted independently of how correctness is established. We therefore prioritize a focused domain in which the meaning and correctness of a generated mathematical model can be examined explicitly.

DC-OPF makes this choice concrete. It requires the LLM to turn the specified network physics and operational requirements into a coherent mathematical formulation. At the same time, its explicit physical variables, well-defined physical relations, and linear constraints make formulation correctness amenable to mathematical verification. In particular, the hard feasible regions of generated and reference formulations can be compared through bidirectional inclusion tests, yielding concrete violating states when they differ. This structure allows us to assess whether the generated model represents the intended physical system and to explain failures through specific mathematical violations. The choice of DC-OPF thus makes both the capability being tested and the evidence used to assess it explicit.

With this probe in place, the workflow experiments serve as inference-time interventions on modeling responsibility. Across four progressively structured settings—full-formulation generation, access to local physical primitives, deterministic constraint construction, and deterministic binding and construction—we examine how the observed failures change as responsibilities are reassigned. Providing correct local physical quantities does not ensure that LLMs assemble correct global equations. Once complete constraint construction is handled deterministically, entity binding and scope become the dominant remaining errors; the final workflow then assigns these operations to deterministic software (Sec. 6.4 and App. H.3). These experiments deepen our understanding of the observed failures by identifying which modeling operations remain difficult despite support for other operations, and which interventions address them. The resulting workflow puts this experimentally supported allocation of modeling responsibilities into practice.

AuthorsOpenReview ↗

Revision Summary

We have revised the manuscript to incorporate the clarifications and additional evidence presented in our rebuttal. The revision clarifies our motivation for using DC-OPF as a mathematically verifiable probe of LLM physical-modeling capability and refines Contribution 4 to emphasize the findings from four inference-time responsibility-allocation conditions. We also add the execution-feedback baseline, a comparison with related benchmarks, and concrete feasible-region violation evidence, while improving figures, terminology, and experimental descriptions. Appendix L.2 presents a benchmark instruction verbatim alongside abridged versions of a passing and a failing response. The core task, evaluator, and main nine-model results remain unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.