acceptodds
Under review as a conference paper at ICLR 2027

TerraWatt: Discovering Hidden World Dynamics Through Irreversible Capital Allocation in a Physically Grounded Voxel Environment

Abstract

Long-horizon agent benchmarks divide into two families. One tests business coherence over simulated months or years, but hands the agent a world whose rules are legible from its prompt. The other withholds the rules, but tests discovery in abstract, deterministic puzzles with no capital, no risk and no delayed payoff. Neither asks whether an agent can infer how an unfamiliar world works and then commit irreversible resources to what it learned. We introduce TerraWatt, in which an agent runs a newly founded renewable-energy developer with $40M of equity on a 2.56 km voxel map. It sites solar and wind plant on land that binds, exports through a capped grid connection into an hourly market that its own output depresses, pays for permits and overruns as they fall due, and can go bankrupt. It is given the goal, its tools and a handbook of priors, but not the site-specific quantities that decide success, such as the wind resource, each vendor's true panel degradation, price elasticity and its own operating costs. These are drawn per seed from documented real-world distributions and recoverable only through costly, delayed measurement. Every exogenous process is action-invariant, so runs are bit-reproducible and agents can be compared on identical worlds, including against a hindsight planner. Running each agent both blind and briefed with the hidden parameters yields a discovery gap that separates failing to find out from failing to plan. In a pilot with three frontier models the gap takes both signs, from −0.57 to +0.11; with the parameters revealed, one model outplans a 112-candidate hindsight search; stated beliefs expose confident errors that performance hides; and the agents found nine defects in the environment that several hundred tests had missed.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agents, long-horizon decision making, world modeling, active experimentation, scientific discovery, sequential decision making under uncertainty, calibration, energy systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n5qCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TerraWatt is an energy-company simulation environment for studying how agents acquire costly information about hidden world parameters and make irreversible, long-term investments. The authors compare Blind and Briefed conditions to distinguish limitations in information discovery from those in planning. The environment supports deterministic replay and paired evaluation, but the current LLM experiments comprise only six runs across three models in a single world.

Critical strengths
  1. The paper studies an interesting problem that combines costly information acquisition with irreversible investment and delayed returns, requiring agents to use what they learn to make consequential decisions.
  2. The environment keeps exogenous processes identical across agents on the same seed and supports deterministic replay, enabling controlled comparisons and detailed inspection of decision trajectories.
Critical weaknesses
  1. The LLM evaluation consists of only six runs: three Claude models, each evaluated once under Blind and Briefed conditions on a single world seed. Without multiple seeds and repeated runs, it is unclear whether the magnitude or even the sign of the discovery gap is stable. This limits support for the claim that the benchmark distinguishes information-discovery limitations from planning limitations.
  2. The two-year evaluation horizon and terminal valuation may obscure the capability the benchmark aims to measure. Median wind construction time is approximately 26.5 months, while unfinished projects receive zero terminal value, structurally disadvantaging strategies that measure before committing. Performance may therefore reflect adaptation to the cutoff and appraisal rules more than effective discovery and use of hidden parameters.

Please also check that the manuscript formatting follows the conference template; the title currently appears not to be bolded.

Questions for authors
  1. Do the discovery-gap results remain consistent across multiple world seeds and repeated runs?
  2. How sensitive are the conclusions to a longer evaluation horizon or a terminal valuation that assigns value to unfinished projects?
Reviewer_Y3iDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TerraWatt, a simulator for evaluating agents that learn hidden environmental parameters while managing a renewable-energy company. Its Blind and Briefed tracks compare the same agent with and without access to these parameters under matched external conditions. The paper presents baseline evaluations, alongside simulator validation and implementation audits.

Critical strengths

1.The paper studies a worthwhile problem. Testing whether agents acquire useful information before making costly commitments goes beyond measuring task completion alone. This is a relevant direction for evaluating long-horizon agents.

2.The examples show that an agent can achieve a good business outcome while making confident estimation errors. These measurements help explain behavior that the final score alone would miss.

3.There is clear attention to reproducibility and simulator validation. The paper reports execution replay checks and comparisons of physical outputs against external references. These are valuable checks for a benchmark whose conclusions depend on the environment’s implementation.

Critical weaknesses

1.The case for TerraWatt’s particular design needs stronger support. Section 2 makes broad claims about entire benchmark families. A focused comparison with the closest environments would better support the claimed gap. The energy setting offers potentially useful mechanisms, including costly measurements, spatial constraints, and delayed returns. However, the paper does not sufficiently establish which mechanisms are essential to the capabilities being tested. Please connect each major design choice to a specific evaluation need. Targeted ablations would help show what the simulator adds beyond domain detail.

2.Section 5.2 interprets a small Briefed–Blind gap as a planning limitation. Yet a small gap can also arise because the Blind agent already learned the relevant facts. Section 6.3 describes precisely this situation for wind conditions. Moreover, Briefed agents receive latent parameters, while the hindsight planner also knows future external outcomes. Remaining below that planner therefore need not indicate poor planning alone. Please distinguish the effect of supplying information from a reliable diagnosis of why an agent fails. The stronger interpretation needs controlled validation.

3.The presentation makes the central contribution and implemented scope difficult to follow. The abstract spends considerable space on financial and implementation details instead of giving a compact account of the simulator and its validated contribution. The introduction then describes features that the final scope paragraph says are not available. Frequent shifts between design goals, experimental findings, and implementation defects further interrupt the argument. Please organize the paper around the research question, simulator design, evaluation protocol, and findings. The abstract and introduction should describe the implemented contribution.

Minor Comments

1.Opening phrases in Contributions and Related Work read like paragraph headings but are not consistently distinguished from the surrounding prose. Please use consistent formatting and improve the transitions between these points.

2.Consider renaming Sections 5 and 6 to “Evaluation Protocol” and “Experimental Results,” or combining them under one section. Also, introduce “defects” explicitly as simulator implementation errors when first mentioned.

Questions for authors

Could you give a concrete example of a capability that the closest existing simulators cannot adequately test? Which specific design choices in TerraWatt make this capability measurable? This would help clarify the contribution beyond introducing a more detailed simulation domain.

Reviewer_opPyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is benchmark paper.

Where an agent purchases information and makes an investment that you cannot reverse, and then runs a simulated energy company.

Blind and brireft tranks comapery hidden parameters versus real parameters, and then the agent evaluation is a 2-year pilot with 3 models on 1 world.

Critical strengths

A particular thing that I like in this benchmark is that it makes information costly, so you need to pay for it, and the decision is irreversible. So if there is also randomness that is action-invariant and it supports really controlled paired comparisons

The good thing is that we have separate belief scores to help distinguish understanding from a successful decision. This is really strong part of the paper.

The bugs that agent discovered, they provide some lessons in benchmark relations too.

Critical weaknesses

The weakness I see is that the discovery gap is not uniquely diagnosing planning failure. It can happen that small gaps can instead reflect easy discovery, as we have the seed 1 result illustrating.

I want to point out that highsighnt r reference also knows future events, which is unavailable to briefed agents.

Only 6 agent runs support evaluation with broken approval grabs, and there are additional status reads that affect comparability.

Finally, using two horizons naturally and mechanically penalizes the measurement, followed by the wind construction.

Questions for authors

The main question is whether the discovery gap remains stable across worlds and repeated runs.

Could the controlled ablations distinguish discovery difficulty from the difficulty of planning? Would you say the paid measurement helps against a match/no-measurement baseline on a sufficiently long horizon?

Finally, can the evaluation be repeated even if we correct the interruption, fix the interaction band, and make no status read from the external side?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.