- Rating
- 3
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how agents can improve incomplete, incorrect, or outdated beliefs about their environment without updating the underlying LLM. EvoBelief maintains testable hypotheses, validates them through subsequent observations or active experiments, and allows supported rules to influence prediction and planning while revising or withdrawing them when counterexamples arise. Experiments across four benchmarks demonstrate improved task performance, cross-task rule reuse, recovery from hidden dynamics changes, and law discovery in simulated scientific environments.
Critical strengths
- The paper clearly separates storing an experience from allowing it to guide later decisions, which makes the contribution more substantial than a conventional memory mechanism.
- The paper provides a reasonably complete evidence chain beyond final success rates.
- This paper is well-written.
Critical weaknesses
- The generality of the framework remains limited by how it is instantiated. This makes it difficult to separate transferable methodological contributions from engineering tailored to these environments.
- Independent validation is central to the paper, but its additional benefit is not clearly isolated. See the questions below.
Questions for authors
- How much of the post-drift recovery in MiniGrid depends on rule withdrawal, rather than renewed exploration or subsequent planning?
- Removing the commitment gate gives results close to the full method in Table 12. Under what conditions does independent validation actually improve decisions?
- Which hypothesis types, conditions, functional forms, and experiment steps are predefined? Which components must change for a new environment?
- Beyond rule-use counts, what evidence shows that retaining rules across tasks actually improves success on later tasks?
5 comments
[2/3] Response to Reviewer 5XzK
[W2, Q2] When independent validation improves decisions
On deterministic MiniGrid, the submitted implementation induced rules in a small, predefined language. Table 12 showed little difference between full EvoBelief (69.6%) and allowing a rule to act on its first supported instance (68.3%). Those rules were rarely wrong, so the gate had little to filter. We revisited this question with a language model proposing the hypotheses. We therefore ran the gate ablation again on MiniGrid and ALFWorld. Shadow retains hypothesis formation and testing while disabling direct use of learned rules. In the no-gate arm, a proposed rule is immediately eligible to guide behavior and loses its authority after two consecutive counterexamples.
| Success, general version, Haiku 4.5 | Shadow | No gate | Full |
|---|---|---|---|
| MiniGrid | 29.8% | 40.2% | 70.9% |
| ALFWorld | 68.7% | 84.3% | 86.6% |
On MiniGrid, the full method outperforms both variants, and the gate already leads before the hidden change, while the dynamics are still unchanged (78.9% vs. 68.6%). On ALFWorld, where the language model already knows most of the dynamics, the two versions solve a similar number of tasks, but the gate makes rule interventions far more reliable. With the gate, 12% of vetoes blocked an executable action, against 44% without it. A language model proposing rules freely is itself a noisy source of hypotheses, and the gate keeps that noise from reaching behavior. Validation therefore improves decisions when acting on a wrong rule is costly, as when the environment departs from what the language model expects or changes over time.
[3/3] Response to Reviewer 5XzK
[Q1] What drives recovery after the change in MiniGrid
Starting from the same state at the switch, disabling rule narrowing and retirement reduced post-switch success from 62.9% to 11.1%. We compared two continuations of each of the 14 test streams. The first is the full method, whose numbers are the test run's own episodes after the change. The second keeps everything, including experiments and new hypotheses, but never retires or narrows a confirmed rule.
| MiniGrid, general version, Haiku 4.5, 280 episodes after the change | Full method | Without narrowing or retirement |
|---|---|---|
| Success | 62.9% | 11.1% |
| Actions driven by outdated rules | 22 | 849 |
| Outdated rules still in force at the end | 1 of 39 | 39 of 39 |
Every stream had confirmed rules that became wrong at the change. With retirement, 38 of these 39 rules lost their authority. Without it, they kept acting and drove 849 actions, most of them based on wrong predictions. Exploration and new hypotheses continued in both versions, so they do not bring recovery back on their own. The shadow version, which learns and explores in the same way but never lets rules act, reaches 22.1% after the change.
[Q4] Whether keeping rules improves later success
Yes. To isolate the effect of keeping rules, we ran a diagnostic test on ALFWorld with no learning during the test. This is not how EvoBelief is deployed, since it normally keeps learning. The agent first learned on the 140 development tasks. We then froze its rules and ran the 134 unseen tasks twice, once with these rules and once without them. Both runs use EvoBelief with the rule store frozen and experiments disabled, not the Frozen WM baseline of the main table. So the only difference between the two runs is the rules kept from earlier tasks.
| ALFWorld, general version, Haiku 4.5, 134 unseen tasks, rules frozen | Success |
|---|---|
| Without rules | 64.9% (87/134) |
| With the rules kept from the development tasks | 73.1% (98/134) |
The kept rules let the agent solve 21 tasks that it misses without them, while it misses 10 that it solves without them. They vetoed 421 drafted actions in 119 tasks, and 22 of these vetoes were wrong. In the main results, the agent starts with no rules and learns during the test, and success reaches 86.6%.
We thank the reviewer again for the constructive feedback and questions. They helped make our work clearer and stronger, and we will add these results and analyses to the revision.
[1/3] Response to Reviewer 5XzK
We thank the reviewer for the thoughtful review, and for recognizing that EvoBelief separates storing an experience from letting it guide decisions!
[W1, Q3] Generality, and what is fixed in advance
EvoBelief is designed to learn how a particular environment works. For example, a MiniGrid agent can be told that it may pick up a key and toggle a door, and that the door is locked. It still has to discover which key opens the door, and whether that stays true later. The knowledge the agent learns should differ from one environment to the next. We never gave it the rules. Our question was whether an agent given a basic description of an environment can discover the hidden rules of how it works.
Across the submitted implementations, a hypothesis specifies a condition, an action or procedure, and a predicted outcome (C, A, Y). Appendix B describes action effects, properties and measurements, multi-step procedures, and conditional constraints. For cross-episode reuse, evidence must span at least two episodes and two bindings. Each benchmark also had a predefined vocabulary for conditions and outcomes, for example four state conditions on MiniGrid, and Reactor Lab considered four law families (constant, proportional, linear, and quadratic). Experiment steps are not written by hand but compiled from each hypothesis, as steps that set up its condition, take its action, and read its outcome. In the submitted versions, this vocabulary, a few task helpers, and the ways learned rules are used (Table 4, Appendix C) change with the environment, while the lifecycle stays the same.
To test generality more directly, we evaluated a version of EvoBelief in which the environment supplies only the goal, the observation text, and the list of available actions. The language model proposes candidate hypotheses directly from observations, without relying on benchmark-specific rules for hypothesis generation. The same procedure tests each rule's predictions and narrows or retires a rule when it is contradicted. The same code and prompts run unchanged on MiniGrid and ALFWorld. All methods were re-run in one shared setup with the same inputs, model (Claude Haiku 4.5), and budget, where each actor writes a short thought and sees the full history of the current episode up to that step, without retrieving transitions from earlier episodes (as in standard ReAct).
On MiniGrid, where the environment departs from what the language model expects, the general version is ahead of all four baselines in this table. On ALFWorld, where the model's expectations are mostly right, it is still ahead of them. Our reply to Reviewer 8uWN also compares it with AutoManual and WALL-E 2.0. These results show that an environment can supply the actions while leaving the relevant conditions and outcomes for the agent to discover. We will add the general version and these results to the revision.