- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores using large language models (LLMs) to construct complete, executable mathematical models of physical systems. Specifically, in this work, full-formulation DC-OPF synthesis is used, aiming to get an LLM to construct a complete executable formulation from a natural-language dispatch instruction and an explicit modeling contract, without a supplied OPF backbone. In the paper, the authors formalize this problem, develop a certificate-based evaluator, and test several LLMs. The results indicate that there are substantive physical-modeling errors, and propose a different pipeline, where the modeling responsibilities are such that LLMs are used to interpret natural language requirements, while deterministic software does the actual construction of the complete DC-OPF. The proposed pipeline improved results (fewer errors).
Critical strengths
- The proposed pipeline results in fewer errors, thus is a viable and useful result and insight for the application of LLMs for this DC-OPF problem.
- Evaluation includes human expertise for validating the input to the LLM, and also the resulting outputs.
- The evaluation also considered different strategies and proposed pipeline to demonstrate the improved performances and analyse the amount of inference tokens (costs).
- The overall high-level story and objective of the paper was clearly presented and, therefore, easy to understand.
Critical weaknesses
- The paper uses LLMs without specific tuning for the given problem, and the results of 818/1062 indicate that 77% of the time the LLM produced correct outputs. What error rate is acceptable in practice, and how do the LLMs compare with other methods commonly used for this modeling?
- The proposed pipeline is hand-engineered and it seems to me that it is expected that it works better. Giving the LLM more opportunity to improve the way it generates the outputs for the full-formulation by allowing tool calls and execution of the code for validation, it would also perhaps lead to better performance. Thus, adding, for example, a fine-tuned LLM for this problem and an LLM with tool calls and ability to execute and test code, for comparison with the proposed pipeline and baseline would be beneficial.
- Motivation to use LLMs in this way (for the full-formulation baseline) is unclear. Why do we want to ask an LLM to write a whole DC-OPF from scratch in practice, when there are existing tools for that?
- The evaluation is strictly binary (pass/fail), so an output with one minor error scores the same as one that is entirely wrong. Partial-credit or severity information would help judge how close the models are to successful modeling.
- Evaluation seems to include single runs of LLMs (Table 1). Multiple runs should be executed with statistics on the outcomes. Also, initial experiments are conducted with 9 models, while later analysis and comparison is done with only 1-2 models, which decreases the generalizability of the findings.
- While the overall high-level understanding of the objective of the paper is clear, it was hard to understand the details of the work. Many abbreviations are not defined (e.g., JDIMP, JDEF). Moreover, adding more explanation on the specific cases used (case30, case118, case300) could lead to better understanding of the work (to those that are not familiar with the PGLib library). Moreover, a lot of the work is shown only in the very long appendices. Figure 1 uses a lot of space for little information. Some figure text in Figure 5 is clipped and unreadable.
- While this is an applied LLM paper, it is likely more valuable to the DC-OPF community than to ICLR. The paper offers practical information for DC-OPF researchers on how to better leverage existing LLMs for their domain, while insights about LLMs are fairly limited.
Questions for authors
Please address the stated weaknesses.
5 comments
Response (2/2)
4. Correctness certification and error diagnosis (W4)
The certificate outcome is binary because certification requires all applicable correctness checks to pass. For feasible tasks, transformed states, objectives and soft epigraphs, and complete hard feasible regions must agree with the reference within the specified numerical tolerances to establish instance-level formulation equivalence. The pass rate reports the proportion of candidates satisfying the required checks.
Our analysis goes beyond this binary outcome to characterize the errors and their mathematical consequences. We review all 244 full-synthesis certificate failures and classify each by its first confirmed error: 90 in physical relations, 49 in entity binding and scope, 36 in constraint semantics, and 69 in program implementation. Bidirectional violation LPs additionally provide concrete witnesses of feasible-region discrepancies. Among 26 formulations rejected by feasible-region checks, 11 have matching candidate-declared, externally reconstructed, and reference optimal values (§4, App. L.2).
5. Repeated runs and presentation (W5, W6)
Appendix H.5 already reports three independent complete runs for GPT-5.6 Luna and Gemini-3.6 Flash under both evaluated conditions. Full-synthesis certificate passes are 53/52/53 for Luna and 78/83/85 for Gemini, directly characterizing run-to-run variability. The nine-model evaluation establishes the breadth of observed failures, while the targeted experiments with fixed models examine specific error mechanisms and interventions.
We will define JDIMP/JDEF at first use, better explain the PGLib network cases, reduce Figure 1, and improve the readability of Figure 5.
Thank you for the detailed response and the additional experiment.
- In your response you clarify the research objective of this work, but together with how the paper currently reads, this framing makes the objective less clear to me. Specifically, it is not clear whether the paper is primarily an applied study of LLMs for DC-OPF with specific DC-OPF findings, a new method (the proposed pipeline) for DC-OPF modeling, or a benchmark for evaluating LLMs. The paper currently has elements of all three, and these would be assessed differently. If the benchmark is a main contribution, the paper would benefit from guidance on how others should use it: a standardized evaluation protocol, how the benchmark can be extended or used, and more explanation for ML researchers without power-systems expertise. It would also need a clearer comparison with existing related benchmarks mentioned in the related work, such as ProOPF and Text2Opt-Bench: how similar the tasks are, which insights are new and which confirm prior findings, and what the benefit is of adding another closely related problem. If the main contribution is the applied study or the pipeline, could you clarify which insights transfer to the broader ML community beyond DC-OPF?
- You frame the main objective of the paper as a test of LLMs’ physical-modeling capability ("organize knowledge of the physical world into reliable foundations for reasoning and decision-making"). Since the modeling contract provides the model variant, conventions, and units, the task seems to test faithful implementation of a specified physical model rather than the choice of how to model a physical system (which laws, approximations, and assumptions apply). Can you clarify this difference and how it affects the claims in the title and conclusions?
- The new experiment is a useful addition. Could you report its number of calls and inference tokens, as in Table 2? Could you also classify the 12 remaining failures using the same four categories as Table 1 (physical relations, entity binding and scope, constraint semantics, program implementation), for consistency and to better understand which error types the feedback helps with? For DeepSeek V4 Flash, feedback reduces failures from 17 to 12, which is a modest difference from a single run, and below the reflection strategy (113/118). Since DeepSeek V4 Flash is among the stronger models in Table 1, there is limited room for improvement. If feasible within the discussion period, running the baseline for GPT-5.6-Luna, which had 65 failures, would show more clearly whether execution feedback meaningfully helps.
As a suggestion for the revision, I would recommend adding examples of the prompts and outputs (prompt instruction, and a successful and failing output). This would add additional information and clarity on the tasks for LLMs, especially for readers outside the power-systems field.
Points 1 and 2 are more important for my assessment, and at the moment I still consider the paper borderline.
Response (1/2)
We thank the reviewer for the constructive comments.
1. Research objective and relevance (W1, W3, W7)
Physical-modeling capability provides evidence of whether general-purpose LLMs can organize knowledge of the physical world into reliable foundations for reasoning and decision-making. We use DC-OPF as a mathematically verifiable probe of this capability. DC-OPF combines substantive physical-modeling demands with mathematically tractable verification, allowing us to distinguish incorrect formulations from merely different representations.
Our full-formulation task requires constructing complete executable formulations under an explicit modeling contract. It directly tests whether LLMs can construct the power-flow equations, nodal balances, unit conversions, and standard constraints that backbone-based tasks already provide. Across nine LLMs, we identify 90 first-attributed physical-relation failures, including 38 missing MW/per-unit conversions, 21 incorrect impedance-base conversions, and 18 branch-direction/incidence-sign errors. Separately, 49 failures concern entity binding and scope (§6.2, App. L.1). These findings expose concrete difficulties in both physical-relation construction and binding requirements to network entities.
Together, formulation checks and source-level error analysis measure the reliability of LLM physical-model construction and identify where it fails. The observed success rate measures formulation reliability; individual models intended for practical use require correctness checks rather than acceptance based on an aggregate success rate. In safety-critical applications such as power-system operation, even a single error in representing physical constraints may lead to unsafe operating decisions.
2. Progressive interventions reveal failure mechanisms (W2)
We agree that deterministic construction is expected to improve reliability. Our contribution is the empirical investigation of which modeling operations cause errors and which errors persist when those operations are handled deterministically.
We use “construction errors” to denote the combined count of failures first attributed to physical relations, constraint semantics, or program implementation. Each failure is counted once.
We progressively introduce local physical primitives, complete constraint construction, and deterministic entity binding (§6.4, App. H.3). Across DeepSeek V4 Flash and GPT-5.6 Luna, construction errors decrease 65 → 25 → 1 across the first three conditions, while entity-binding/scope errors remain 17, 18, and 22.
These results reveal two distinct difficulties: correct local physical coefficients do not guarantee correct assembly of flow and balance equations; removing equation-construction errors does not eliminate incorrect entity grounding. The proposed allocation is informed by these observed failure mechanisms rather than merely by the expectation that a deterministic backend performs better.
3. New execution-feedback baseline (W2)
We additionally evaluate DeepSeek V4 Flash with execution feedback and two rounds of revision. This condition achieves 106/118 certificate passes, compared with 101/118 for single-call synthesis, 113/118 for reflection, and 112/118 for two builders plus an integrator. We will include this experiment in the revised paper.
The 12 remaining failures include six incorrect state updates, four missed rejections, one Python call error, and one nodal-balance sign reversal. In
case118_A_006, the generated model matches the reference optimal objective, yet a violation-LP witness identifies a 102 MW nodal-balance discrepancy.Execution and solver feedback can identify executable-code failures without establishing the correctness of the underlying physical formulation. This provides additional evidence for the distinction between successful execution and faithful physical modeling.
Our evaluation deliberately examines general-purpose LLMs without task-specific tuning. Fine-tuning investigates specialized adaptation, whereas these experiments examine existing modeling capabilities and the effects of inference-time intervention.