- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates: when an LLM agent omits derived conclusions that remain exactly derivable from retained facts and rules in the current-state representation of a multi-step tool-use task, how does this affect agent behavior? The paper defines two symmetric conditions: Explicit (derived conclusions are directly provided in the state representation) and Derivable (those conclusions are omitted when they remain exactly derivable). Crucially, omitted information under the Derivable condition is formally verified to be reconstructible via an exact reconstruction check, ensuring the experiment directly answers the titular question rather than testing information loss. Key findings: in TextWorld, the success gap is 61.7%; in ALFWorld, 16/100 vs 1/100. Content controls show the effect comes from the derived conclusions themselves, not from extra text or repeated premises. Action-level analysis from shared states finds all 31 first divergences in TextWorld favor Explicit. The gap weakens or disappears when the model has sufficient pre-action computation (thinking mode or free-form notes). System-side deterministic derivation achieves 58/60 tasks with approximately 60% of the tokens and half the calls, representing the efficiency-optimal design.
Critical strengths
- Precise problem definition with formal guarantees.
The exact reconstruction check (Equation 2) is the paper's core methodological contribution: a task-state pair enters the behavioral comparison only when the Derivable interface can be deterministically rebuilt into the Explicit interface using retained facts and a fixed public rule set. This formally operationalizes the distinction between "logically derivable" and "actually omitted," making the experimental conclusion directly address the paper's question rather than confounds.
- Large and robust effect size.
61.7% points in TextWorld is a highly significant effect, which holds across content controls: neutral fillers (P0–P50) do not reproduce it, and repeated premises (P_REPEAT) do not reproduce it, isolating the source to the derived conclusions themselves rather than token count or context length.
- Complete multi-level analysis from task to action.
The paper uses deterministic transitions at temperature zero in TextWorld to analyze the first action divergence from shared starting states (all 31 favor Explicit), and uses pre-selected shared states in ALFWorld to show the effect does not require prior trajectory divergence. This multi-granularity analysis substantially strengthens the causal story.
- Insightful pre-action processing experiment.
The preparation experiment in Section 5 (free-form notes C vs. complete reconstruction R vs. direct action D) reveals an interesting asymmetry: under the same preparation budget, free-form notes outperform requiring explicit complete reconstruction. This finding goes beyond a binary "provide or not" comparison and delivers practical guidance on how to exploit derivable information.
Critical weaknesses
- Environments are highly controlled.
TextWorld and ALFWorld are gamified text and simulation environments whose "derivation rules" form a pre-specified, fully formalized, deterministic rule set. In real-world agent tasks such as code execution, database queries, API calls, the scope of derived information, the completeness of the rule set, and whether the "exactly derivable" condition holds at all are far less clear than this paper assumes. Whether the exact reconstruction check is even verifiable in real deployments is an important question the paper does not address.
- Narrow model coverage with inconsistent configurations across experiments.
The main quantitative results come from Qwen3.8-27B with thinking disabled. ALFWorld adds Gemma and DeepSeek for action-level comparisons, but task-level success comparisons cover only two models, and the thinking mode is toggled differently across experiments. Without systematic control over model scale, training data, and instruction tuning, the conclusion that "the effect varies across executors" is hard to bound.
- No analysis of which derivations are hardest.
The paper focuses on the binary presence or absence of derived conclusions, but does not analyze which properties of a derivation (such as chain depth, number of premises, counter-intuitive rules) predict failure under the Derivable condition. Such analysis would substantially deepen understanding of why LLMs fail to perform these derivations on demand, and would create a more direct connection to the rule-reasoning literature the paper cites (Clark et al., 2020; Dziri et al., 2023).
- ALFWorld experiments are asymmetric with TextWorld.
TextWorld provides complete task success rates, action-level analysis, and preparation experiments. ALFWorld provides only task-level success rates for two models and action-level comparisons for three models, with no preparation experiment. This asymmetry weakens the cross-environment generalization and leaves open whether the preparation-experiment findings extend beyond TextWorld.
- Uneven statistical reporting.
TextWorld results use exact sign tests; ALFWorld action-level results include paired tests with confidence intervals; but some results, particularly for Gemma and DeepSeek in ALFWorld and for Qwen with thinking enabled, are reported as point estimates only with qualitative uncertainty labels such as "uncertain." For a paper whose primary contribution rests on precise experimental control, the statistical reporting should match that standard throughout.
Questions for authors
-
What is the pass rate of the exact reconstruction check? How many tasks or states were excluded because they did not pass it, and what were the predominant causes of failure (rule ambiguity, incomplete state, other)? This number directly bounds the scope of the conclusions.
-
Appendix C.3 notes that all 60 TextWorld tasks have the same maximum derivation depth. Does the derivation complexity vary at all across those tasks? If all derivations are single-step, how should readers interpret the paper's generalizability to deeper derivation chains?
-
In the preparation experiment, conditions C and R share the same "preparation budget." How is this budget operationalized, eg, in tokens, reasoning steps, or something else? And how many reconstruction steps can C complete within that budget on average? If C cannot complete a full reconstruction within the budget, then C < R may simply reflect that full reconstruction is more expensive than free-form notes, rather than that partial reconstruction is worse.
-
Qwen3.8-27B with thinking disabled is the primary executor for the main results. Was this configuration chosen specifically because the effect disappears with thinking enabled? If so, does this imply that the effect is largely absent in modern reasoning models under their default settings?
-
The paper's core condition "exact derivability" is verified offline with a deterministic checker. Is this check feasible at inference time in real systems? For cases that are "mostly derivable" or "approximately derivable," how should practitioners apply this paper's conclusions?
2 comments
Thank you for the follow-up. We agree that with thinking enabled both conditions reach ceiling on the TextWorld tasks (60/60 vs. 59/60 at 4K, 59/60 vs. 59/60 at 16K), and the revised paper states this.
Two results in the updated PDF bear on practical significance. First, the mitigation is paid for in generated tokens. On the same tasks, Explicit without thinking completes 57/60 and Derivable with thinking 59/60, while Derivable generates about 35 times the completion tokens per task (8,691 vs. 246 at the 16K limit) and 1.24 times the total tokens. Stating the values adds about 6% to the first prompt. Second, reasoning was not always sufficient. At the first ALFWorld decisions with thinking enabled, Derivable ran out of tokens before returning an action in 39 of 100 states, against 9 for Explicit, at 16K, and in 22 against 5 at 32K. In BabyAI at 8K, 19 Derivable and 10 Explicit executions stopped before returning an action.
Omission was therefore safe only when the executor reasoned and its generation limit did not bind, which is known only at run time (Section 6, Appendices C.6 and C.9).
Thanks for the detailed responses. The clarifications address some of my questions, but the results with thinking enabled further suggest that the effect is largely mitigated by sufficient reasoning computation, which limits the practical significance of the findings. I will maintain my original score.