- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TerraWatt is an energy-company simulation environment for studying how agents acquire costly information about hidden world parameters and make irreversible, long-term investments. The authors compare Blind and Briefed conditions to distinguish limitations in information discovery from those in planning. The environment supports deterministic replay and paired evaluation, but the current LLM experiments comprise only six runs across three models in a single world.
Critical strengths
- The paper studies an interesting problem that combines costly information acquisition with irreversible investment and delayed returns, requiring agents to use what they learn to make consequential decisions.
- The environment keeps exogenous processes identical across agents on the same seed and supports deterministic replay, enabling controlled comparisons and detailed inspection of decision trajectories.
Critical weaknesses
- The LLM evaluation consists of only six runs: three Claude models, each evaluated once under Blind and Briefed conditions on a single world seed. Without multiple seeds and repeated runs, it is unclear whether the magnitude or even the sign of the discovery gap is stable. This limits support for the claim that the benchmark distinguishes information-discovery limitations from planning limitations.
- The two-year evaluation horizon and terminal valuation may obscure the capability the benchmark aims to measure. Median wind construction time is approximately 26.5 months, while unfinished projects receive zero terminal value, structurally disadvantaging strategies that measure before committing. Performance may therefore reflect adaptation to the cutoff and appraisal rules more than effective discovery and use of hidden parameters.
Please also check that the manuscript formatting follows the conference template; the title currently appears not to be bolded.
Questions for authors
- Do the discovery-gap results remain consistent across multiple world seeds and repeated runs?
- How sensitive are the conclusions to a longer evaluation horizon or a terminal valuation that assigns value to unfinished projects?