Weaknesses
1. Substantial problems with clarity limit assessment of the contributions
I found the paper exceptionally difficult to follow due to several presentation issues, including but not limited to: convoluted language; dense combinations of undefined terms and jargon; explanations that too often defer details to the appendices; historical experiments and protocol revisions interleaved with the principal results; unclear interpretation of findings; and abrupt introductions of new concepts that are never defined in the main text.
In my assessment, these issues are substantial enough to warrant rejection, as they prevent a reliable evaluation of the methods, results, contributions, and the paper's overall significance.
I provide representative examples at the end. I strongly encourage the authors to revise the manuscript extensively, especially if substantive parts of the text was written with the assistance of AI.
2. Limited realism and unclear relevance of the benchmark
The primary benchmark consists of relatively simple, synthetic grid-world environments. Although such environments can be useful for isolating specific capabilities, the paper does not sufficiently establish how its findings relate to the challenges faced by LLM agents in more realistic settings.
The additional evaluations involving Kirby's Dream Land, Crafter, and Boolean workflows provide some diversity, but their introduction and experimental methodology are insufficiently explained. Furthermore, it is unclear to what extent these experiments establish generality beyond relatively constrained and artificial environments.
This concern is relevant because several LLM agents perform poorly even when provided with truthful action names or additional information about the action effects. It is therefore difficult to determine whether the observed failures reveal limitations in action-model acquisition and planning, or instead arise from the particular environment design, prompting strategies, and agent implementations.
I would encourage the authors to better motivate the relevance of the proposed environments and discuss what conclusions can reasonably be generalized from them.
3. Unclear agent memory and retention of information between phases
An important aspect of the benchmark is that agents are expected to acquire information during calibration and use it during the subsequent execution phase. However, the manuscript does not adequately explain how this information is preserved and made available to the LLM agents.
My understanding is that at least some implementations rely primarily on the conversation history containing previous observations, actions, and feedback. If this is the case, context-window limitations and history truncation could directly affect the agents' ability to use information acquired during exploration (which seems to be acknowledged only in appendix J)
If informative observations are removed from context before or during execution, poor performance may reflect loss of previously acquired evidence, rather than difficulty identifying action dynamics or planning effectively. This is an important potential confound given the paper’s objective of distinguishing action-model acquisition from its subsequent use.
The authors should describe the memory mechanisms more clearly in the main text and explain their implications for interpreting the results. If agents rely solely on a truncated conversation history, evaluating alternatives such as persistent summaries of observed action effects would help determine how much the reported failures reflect loss of exploration evidence rather than difficulties in action-model acquisition or planning.
4. Missing methodological details and unclear status of experimental components
Several experimental components are introduced without sufficient methodological detail, making it difficult to understand their role in the study and assess the corresponding claims. For example:
- The paper introduces L2 as part of its diagnostic framework but repeatedly acknowledges that it was not evaluated. Could the authors explain more directly why an unevaluated condition is included in the main diagnostic framework and which parts of the proposed decomposition remain untested without it?
For reference, here are some passages:
Section 3.2:
"L2 is specified but not evaluated here, so no L2 result is claimed: the archived runs cover two models only, and the summary artifact they injected was not preserved"
Section 4 (experimental setup):
"L0 is the main anonymous setting, L1 and L3 are targeted probes, and L2 is specified but not evaluated.
The whole section "H.3 L2 Is specified but not evaluated"
The L2 protocol (systematic action tests in Phase 1, followed by a structured observation summary for execution) is part of the design but is not evaluated in this paper, and no L2 number is claimed anywhere in the main text. Three reasons, none of them a result.
Could the authors clarify the section "H.3 L2 Is specified but not evaluated"? For instance, how should readers interpret this? How important is this to the paper's claims?
First, the protocol as implemented and as specified diverged: the archived runs use a Phase-1 round count that the specification does not license, so what ran is not the treatment L2 names. Second, the summary text actually injected in those runs has no preserved artifact—the code path that builds it changed afterwards, so byte-identity with the current build_phase1_summary cannot be established; the protocol may be described as specified, not as executed.
- Insufficient specification of the Kirby’s Dream Land experiment. This environment appear as part of Figure 1 summarizing major results and the results in Section 4.2. However, I could not find a sufficiently detailed specification of the environment, the complete action interface, the exact prompts, or the experimental protocol.
Given its prominence in the presentation and its role in supporting claims about the generality of action-naming effects beyond grid-world environments, these details seem important for interpreting and assessing the results. Could the authors provide a more complete and reproducible description of this experiment?
Examples of clarity issues
Section 3.2
This passage defines the metrics for interpreting the results. Could the authors clarify it and move any necessary details from the appendix to the main text? For instance, what does it mean to "apply fixed rules"? What is an "archived forward model"?
Exploration coverage (EC) measures the fraction of actions tried; P1 is Phase-1 length. We report three diagnostics: calibration-accuracy (CA) proxy (Table 1) applies fixed rules to each action’s first observed transition; stated-effect accuracy scores the model’s answers to effect questions, with prior/post-interaction comparisons; and held-out prediction accuracy tests archived forward models on unseen state–action contexts. Stated-effect and held-out probes cover the subsets in Appendix H. We report success rate (SR); action counts enter the efficiency indices in Appendix N.
Section 4
We use the diagnostic interventions of Section 3 as operational diagnostics for Explore, Interpret, and Plan: L0 is the main anonymous setting, L1 and L3 are targeted probes, and L2 is specified but not evaluated (Appendix H.3). We compare heuristic baselines (Random, Systematic, Bayesian), Qwen3.5 across sizes and modalities, and five further models (Table 1). Oracle+BFS supplies a known-dynamics reference; Forward-Model Babbling (FMB) variants appear in Section 5. We report success rate (SR) together with calibration efficiency (CE/CEcond), the trajectory-only CA proxy, exploration coverage (EC), and Phase 1 length (P1; definitions in Section 3 and Appendix H).
Could the authors clarify:
- What is "Oracle+BFS"?
- Why "L2 is specified but not evaluated" and the relevance of this passage?
Section 4.1 (main results)
Interpretation of results in this section are limited because:
- Lack of clarity of metrics as shown in excerpt from Section 3.2 above
- The implementation details of baselines (Random, Systematic, and Bayesian) have no substantive explanation in the manuscript
- Calibration Efficiency (CE), bold-highlighted in Table 1, is not formally defined until Appendix N, which is never directly referenced to explain the metric
Exploration strategy is one contributor: a Bayesian baseline over permutation matrices (SR 22.7%) and a Systematic one-test-per-action heuristic (17.7%) beat every open-weight vision-language model (VLM) under image input (best 15.7%),
What are "permutation matrices"?
"as the model-output-based replacement for the trajectory-only CA proxy, held-out next-state prediction reaches 0.71–0.89 against 0.05–0.08 for controls"
What is "trajectory-only CA proxy"? How does it connect to the (vague) description of CA in section 3.2?
Section 4.2
Several conditions and terms are introduced without adequate explanation. Several passages combine experimental results, statistical details, and interpretation in ways that make their intended conclusions difficult to identify.
We rerun Adapt and Spatial (50 envs each; GPT-4o and Gemini 3.1 Pro, image+text) under three labelings: anonymous (AlienBody default), named-true (labels match effects), and named-misleading (every label wrong). A familiarity-only control helps only modestly (Appendix L)
What is a "A familiarity-only control"? What it means for it to "helps only modestly"? Why is this relevant here?
"The effect is large and model-specific, and it runs in both directions (Figure 2(a); paired bootstrap CIs; environment-level exact two-sided sign tests count each environment once, pooled-counting values in Appendix L.1)."
What what are "pooled-counting values" and why are they relevant here? Could the authors clarify how to interpret the "both directions"?
"On Spatial, true labels raise GPT-4o's SR from 25% to 40% (+15pp, 95% CI [+1, +29]; p=0.071, a direction rather than an established effect)"
What do the authors mean by "a direction rather than an established effect" and why? Is this because of the p-values (also undefined)? What threshold was chosen to determine statistical significance?
Could the authors clarify the conclusion to derive from this?
The design is floor-bound rather than an equivalence test, since at 1–2% SR a drop has little visible room (the floor truncates the downward direction) while the observed upward differences are also small. It does locate where the name prior stops being measurable: for a model with no room to move, label manipulations leave no trace.
The Kirby's Dream Land is first and abruptly introduced here without any context:
The same confound appears outside the grid, but not always in the same direction. On Kirby’s Dream Land (Game Boy via PyBoy; qualitative, three runs per condition, below the preregistered paired target), GPT-4o plays 300 visual steps under the three labelings (Table 2, left): named buttons (whose descriptions are supplied) reach 8.7 checkpoints (scroll 20.7), category hints (scrambled labels) yield partial progress (scroll 8.7) with more damage, and anonymous IDs leave every run at spawn (max scroll 1.0).
Could the authors clarify:
- What is a preregistered paired target?
- PyBoy?
- How are the three labelings determined in this environment?
- (scroll a.b)?
- What "leave every run at spawn" means?
- How these results connect to the previous passages?
On Crafter Hafner (2022), already flagged for leaking action names in Appendix Table 4, we removed the glosses and re-ran the contrast under a fully shared protocol (17 actions, 20 paired seeds, 150-step cap; Table 2, right).
Could the authors clarify:
- What "already flagged for leaking action names" means? Does this compromise the whole evaluation? Why is it relevant here?
- What it means to "the glosses and re-ran the contrast" under a "fully shared protocol"?
The v1 ordering inverts: true names yield 0.10 mean achievements against 1.85 (anonymous) and 2.45 (category rank labels); named-anonymous -1.75, bootstrap 95% CI [-2.25, -1.30], exact Wilcoxon p=9.7×10-5; category vs. anonymous is within noise (+0.60, p=0.09). The post-episode probe reads named 0.79 > category 0.36 > anonymous 0.12, and named runs try only 4.3 of the 17 actions against 13.3 for anonymous (Table 2, right; per-action mix in Appendix L). The earlier v1 advantage, from a prompt that supplied effect descriptions and gave no condition history, is not reproduced here (Appendix L). Thus, higher stated effect accuracy coexists with narrower action coverage and lower task progress. This is consistent with label-dependent exploration but not established here.
Could the authors clarify this discussion? Also, could they explain what is "v1" and move it from appendix L.2? What does it mean: "the v1 ordering inverts" and "the earlier v1 advantage"? Why this matters?
Section 4.3
Unclear experimental conditions and terminology. Terms such as "complete prose definitions" are introduced without sufficiently explaining what information is supplied to the agents or how it differs from the preceding conditions.
More fundamentally, the very low performance observed on F4, even when action mappings or operator definitions are supplied, raises questions about what capability the experiments are measuring. Is the difficulty primarily due to the complexity of the relational operators, limitations of the LLM-agent implementations, or planning itself? The authors should more carefully distinguish these possibilities.
Abrupt introduction of additional experiments. For example, the paragraph beginning:
"In the L3 condition of the planning wall (Figure 3) the mapping reaches the agent as effect names inside the state, with no operator definitions, no oracle, and no episode history. Asked to predict a single transition on exactly that information [...]"
introduces a one-step transition prediction experiment immediately after discussing Figure 3, which reports end-to-end success rates.
The relationship between these experiments, their different evaluation settings, and their respective roles in supporting the paper's claims are not sufficiently established. The reference to Figure 3 further complicates the presentation because the subsequent results concern a different measurement.
Similarly:
"In a separate registered end-to-end study, supplying the full operator definitions raises DeepSeek V4 Flash's success from 5.0% to 12.5%, while for Qwen3.5-35B-A3B no significant change is detected [...]"
introduces another experimental comparison without adequately explaining its setup or connection to the preceding results.
The gap accompanies distinct behavioral patterns—high action entropy with 26% oracle-plan agreement for Gemini 3.1 Pro, degenerate repetition for a local Qwen model (Appendix L.11)—yet success collapses in both;
No context about action entropy and again, deferring to appendix.
Section 4.4
The three principal experimental conditions (Draft, Oracle, and Program) are only briefly described in the main text. For example, what exactly constitutes a "persistent draft"? What information does it contain, how is it updated, and how does the agent use it? I could not find a sufficiently concrete explanation of its intended contents. Likewise, the operational differences between conversational oracle access and the programmable interface should be made explicit before presenting their comparative results.
"All arms share the task's real-action limit, and oracle arms share its query cap (1,024 on Relational grids; 4,096 on workflow)."
What is a query-cap?
Table 3:
"BFS (J5): 400/400 per task under the same oracle cap."
BFS (J5) is first introduced in the caption of this table and it seems to be only partially explained in Appendix H.7. Could the authors further explain what it means and why is it relevant here?
"The conversational arms run under linear history scaffolding; tree-search agent architectures Zhou et al. (2024a); Hao et al. (2023) are not among the arms evaluated here."
What exactly is meant by "linear history scaffolding"? Why is the absence of tree-search architectures relevant to interpreting the experiments? The authors should explain these considerations rather than introducing them without context.
Section 5
Several undefined terms, appendix deferrals, and vague definitions, in my perspective, compromises all the interpretation of results in this section, despite its potential relevance to the paper's main contributions.
Here is the first time the authors introduce "FMB", as follows:
"We call this plan-over-model pattern Forward-Model Babbling (FMB): babble to identify action schemas, then plan with BFS over a ForwardModelSimulator that applies typed effects—translate, rotate, relational, composite, temporal—without calling the VLM (Appendix K)."
This description introduces several concepts without sufficiently explaining the actual procedure. What constitutes "babbling" in this implementation? How are action schemas represented and inferred? What does the ForwardModelSimulator do, and which components are learned versus predefined?
Supplying a lookup primitive does not by itself close the gap: across one generation on the same relational split, interface only arms stay at 0–4%, a conversational next_state tool at 10–26%, and a Python read-eval-print loop (REPL) at 90–100% (Appendix H.6; unmatched budgets, rematched in Section 4.4).
Could the authors clarify what exactly these are and how they relate to the section's purpose?
- the "lookup primitive"?
- a conversational next_state tool at 10–26%
- Python read-eval-print loop (REPL) at 90–100%?
- (Appendix H.6; unmatched budgets, rematched in Section 4.4).
Also, several important concepts are introduced without adequate explanation in the main text, including:
- "Public operator vocabulary"
- "Enumerate+Verify"
- "FMind"
- "Active FMB"
- "Exact-assignment search"
The notation (n) and (|H⋆), as well as the meaning of "perfect-score ties," should be clearly defined and explained in the main text.
"At n=8 Relational it reaches 90%: remaining failures are perfect-score ties (|H⋆|>1), not verification misses."
"Active FMB (AFMB) selects disagreement probes that split H⋆, lifting the same split to 100% and Temporal Tier-M from 90% to 100%; ablations without information-gain selection fall back to 90–92%"
At n=8, LLM proposers stay weak even with the same verifier (GPT-4o 2%; DeepSeek V4 Pro 6%/4%): verification helps when a strong search proposes candidates, but does not substitute for one. At n=12, an exact-assignment search still reaches 56% while FMind stays at 0%
Unclear LLM-based model induction procedure. In particular, the paper should clearly explain what information is provided to the LLMs when they infer action models, what output they are expected to produce, how candidate models are verified, and how verification feedback is incorporated.
The final paragraph abruptly introduces training as a "complementary stress test," but does not adequately explain its motivation or how it relates to the preceding methods.
The discussion also introduces terms such as "emission," "ground-truth injection," and "verification-filtered self-training" without sufficient explanation.
"the bottleneck is emission, not planning—the same ground-truth injection yields 100% while a content-permuted schema returns 2%"
"verification-filtered self-training does not internalize the search signal on Relational either"
Section 6 — Conclusion
Overall section could be use much less vague language, and explicitly connect to the results observed (esp. lines 479 to 485).
Introduction
Consider adding more specific language and context about the benchmark before describing results; I was only able to parse the content after line 077 after reading the paper several times.