acceptodds
Withdrawn from ICLR 2027

Prompt-Attributable Feasibility Violations of Medium-Weight LLM-Based Control of a Cascaded Five-Reservoir System

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When an LLM is tasked to control a complex real-world problem, a physically infeasible control action is usually projected onto the feasible operating range by the environment, allowing the simulation to continue. Yet, this can mask repeated constraint violations behind strong aggregate performance. Here, we investigate prompt-attributable feasibility failures in a twenty-year, non-resettable, multi-reservoir hydrological testbed, using two LLM families – DeepSeek-R1-Distill and Gemma 4 – as controllers making sequential operational decisions under physical constraints. Empirically, despite achieving near-optimal aggregate performance, the LLM agents violate physical constraints in 10 to 28% of decisions. We show that these violations are systematic rather than stochastic and arise from two reproducible, prompt-attributable reading failures. The first, constraint anchoring, occurs when a field specifies a maximum output limit while explicitly excluding an incoming quantity. DeepSeek-R1-Distill interprets this limit as total availability and systematically under-commits, whereas Gemma 4 does not. The second, forecast presence bias, occurs when a predictive field is introduced into an observation whose other fields describe currently available quantities, causing agents to spend the predicted quantity as if it was immediately available. The effect is driven by forecast presence, rather than magnitude: scaling the forecast to an unambiguous underestimate still triggers violation rates similar to the unscaled forecast case, while removing the forecast drops up to 87% of the violations. We investigate both mechanisms with a five-step attribution protocol – open-loop replay, single-element perturbation with the remainder byte-identical, failure-rate comparison, domain-stripped simplified replay, and token-preserved identity substitution. The results isolate structural misreading from domain reasoning. Constraint anchoring survives complete removal of domain vocabulary, and is eliminated by a single arithmetic clarification. Conversely, forecast presence bias disappears in domain-stripped simplified replay, but returns intact under identity substitution, so neither violation is semantic. Our findings therefore show that the observed feasibility violations arise not from hydrology semantics or the specific benchmark, but from a specific model family's anchoring tendency and a general disposition to remain optimistic even with an underestimated forecast.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
closed-loop control, physical feasibility, LLM agents, prompt anchoring, prompt attribution, constraint satisfaction, agentic evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sGBzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies LLM-based control of a five-reservoir system over a twenty-year simulation. DeepSeek-R1-Distill and Gemma models choose monthly water-release allocations, while the simulator clips requests that violate physical limits. The authors find that high hydropower performance can coexist with frequent infeasible requests. They replay past decisions and edit individual prompt elements to study release-limit misinterpretation at Kafue Gorge Lower and forecast-related over-commitment at Kariba.

Critical strengths
  1. The paper evaluates the LLM's requested releases before the simulator clips them. This exposes repeated infeasible decisions that aggregate hydropower performance alone would miss.

  2. The authors replay decisions and change individual prompt elements to examine the source of the errors. The comparisons provide useful evidence that release-limit wording and forecast information affect the controllers' decisions.

Critical weaknesses
  1. The Kariba analysis does not clearly distinguish forecast misinterpretation from errors caused by uncertain inflow. The controllers do not know the realized natural inflow when making a decision, and the supplied forecasts are systematically too high. Removing the forecast could lead to more conservative releases even if the model understands when that water will arrive. The scaled-forecast tests show that forecasts affect decisions, but do not establish that the models treat future water as currently available.

  2. The prompt changes are tested mainly by replaying selected decisions, not by running new closed-loop simulations. Fewer violations in these one-step tests do not show whether the changes improve long-term feasibility or hydropower performance, since different releases would change later reservoir states. The study also uses one reservoir system and two model families. The broader claims about LLM control are not supported by tests beyond this setting.

Questions for authors
  1. At Kariba, how do you distinguish a reasoning error from a release request that becomes infeasible because actual inflow differs from the information available at decision time?

  2. What evidence distinguishes treating forecasted inflow as immediately available from simply making more optimistic release decisions in response to the forecast?

  3. Were the revised prompts tested in any full closed-loop simulations? If not, how should the improvements from one-step replay be interpreted for long-term operation?

Reviewer_X7dROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper tests whether LLMs can control a complex water-reservoir system over a long horizon. The authors use six open-weight LLM configurations to control five connected reservoirs in the Zambezi River system over 20 years. The main point is that the final hydropower performance can look good even when the LLM often proposes physically infeasible actions. The environment clips these actions before applying them, so the simulation can continue. The models achieve hydropower performance close to the EMODPS reference, but 10–28% of their proposed actions are infeasible. The paper then studies two recurring failure cases. At KGL, DeepSeek models appear to misread the maximum-release limit by not properly accounting for incoming inflow. At Kariba, adding forecast information makes models request more water than the system can provide. Overall, I think the main observation is useful: looking only at final reward can hide unsafe model behaviour. However, I am less convinced by the explanation of the forecast result and by the claim that the proposed prompt changes solve the long-horizon control problem.

Critical strengths
  1. The paper makes an important distinction between good final performance and safe model behaviour. The models get close to the reference hydropower performance, but the environment has to correct many of their proposed actions. This matters because it shows that reward alone is not enough for evaluating LLM controllers.
  2. The authors try to understand the cause of failures, and the replay and prompt-change experiments are a good idea. I appreciated that the paper doesn't stop at saying the model fails, but instead tests which part of the prompt relates to the failure.
Critical weaknesses
  1. I am not fully convinced by the forecast explanation. The experiments show that adding the forecast changes the model’s actions. But I do not think they fully show that the model confuses future water with current water. The model may instead trust the forecast too much, ignore uncertainty, or simply become more aggressive when it sees an additional inflow-related number. This is important because the forecast effect is one of the main claims of the paper.
  2. The proposed fixes are not tested in a new full closed-loop run. The paper shows that changing the prompt reduces failures in replay experiments. But if the prompt changes one decision, it may also change future storage, future prompts, and future failures. I would like to see the full 20-year simulation rerun with the changed prompts. Without this, I am not sure whether the interventions really improve the controller or only improve selected decisions from the original trajectory.
Questions for authors
  1. Can the authors explain the timing within one month more clearly? In particular: when does the model observe storage, when does forecasted inflow arrive, when does real inflow arrive, and when is the release applied?
  2. Why is the Kariba result evidence that the model treats forecasted water as already available? Could this instead be explained by over-trusting an uncertain forecast?
  3. Can the authors run the whole 20-year simulation again with the KGL clarification and with the Kariba forecast modification? I would like to see both hydropower performance and feasibility results.
  4. How often does the safety wrapper change the model output, and how large are these corrections? Since the final score is measured after clipping, this would help show how much of the good performance comes from the LLM itself.
Reviewer_9AzVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the reliability of LLMs as controllers in a long-horizon physical system, focusing on feasibility violations that may be obscured by aggregate task performance. The authors evaluate six configurations from the DeepSeek-R1-Distill and Gemma 4 families in a twenty-year simulation of a five-reservoir cascade. Despite achieving 93.7%–97.2% of the hydropower performance of the EMODPS reference policy, the LLM controllers generate physically infeasible commands in approximately 10%–28% of their decisions.

Critical strengths

The paper addresses an important problem in the safe deployment of LLM-based agents.

Critical weaknesses

1.Missing simple control baselines make performance hard to interpret. The environment includes a safety wrapper that clips infeasible actions. It is therefore unclear how much of the reported performance comes from LLM decision-making rather than from the wrapper’s corrections. The authors should compare against simple non-LLM baselines, such as an always-100%-allocation policy, a rule-based controller, and a greedy feasible-release policy, all under the same safety wrapper.

2.The proposed failure mechanisms are not sufficiently distinguished from alternatives. For constraint anchoring, the reduction in violations after adding an explicit arithmetic formula suggests that clearer instructions help, but it does not show that the model originally relied on the specific incorrect calculation hypothesized in the paper. For forecast presence bias, the results are also consistent with models making more optimistic or risk-taking release decisions, rather than treating future inflows as currently available water. Downscaled forecasts still inducing violations do not fully distinguish these explanations.

3.The prompt interventions are not validated in long-horizon closed-loop control. The intervention experiments are mainly open-loop replays. In a closed-loop reservoir system, actions change future storage, inflows, and decision states, so single-step reductions in violations may not translate into better long-term reliability or performance. The authors should evaluate the original, constraint-clarified, forecast-modified, and combined prompts in full 20-year closed-loop simulations under identical hydrological conditions, reporting both feasibility violation rates and aggregate operational performance.

Questions for authors

More controlled experiments are needed—for example, keeping forecast values identical while varying only whether the water is described as currently available, expected later, or uncertain, and asking models to separately estimate current and future resources.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.