- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies when natural-language procedural guidance helps or hurts frozen LLM agents. The main experiment uses a “search-execute” guidance in ALFWorld with Qwen3-32B-AWQ with three information states: ordinary observation, an equal-length irrelevant fact, and the target object’s location. Guidance improves success by +13.00pp and +22.33pp in the first two states but reduces it by −12.67pp when the location is known.
Critical strengths
- The controls are carefully designed. Token-matched control instructions, tokenizer-matched facts, and a destination-fact condition attempt to separate the effect of guidance content from length, format, and the mere addition of true information.
- The distinction between instruction compliance and task success is practically useful for evaluating agent guidance.
Critical weaknesses
W1. The main finding is expected, and the evidence is too narrow to support a general framework.
The main result is that telling an agent to search for an object whose location it already knows makes it waste steps and fail more often. This is close to what one would expect. Moreover, the evidence comes almost entirely from one setting: one instruction (search-execute), one environment (ALFWorld), and one type of added information (the target location). The BrowserGym experiments do not broaden this, since they mostly show no effect (Appendix A.1) or test a different instruction–information pair (Table 4). This is not enough to show that task fit and remaining need determine the usefulness of guidance in general. There is also a simpler explanation that the paper does not discuss: the model follows the instruction even when the information in its context makes the instruction unnecessary. The paper itself cites prior work showing that extra context does not always help (Sec. 2). The authors should explain what their framework adds beyond this single example.
W2. The paper’s explanation for why guidance hurts is not supported by its own experiments.
The paper explains the harm as follows: when the location is known, the guidance still makes the agent search, and these extra steps use up the step limit before the task is finished (Eq. 5). Two of the paper’s own experiments do not fit this explanation.
-
Larger step limit: If running out of steps were the main cause, doubling the limit from 20 to 40 should reduce or remove the harm. It does not (Fig. 5b). The paper then suggests another cause, that the extra steps lead the agent into worse states, but this is not tested.
-
Direct evidence that the guidance causes extra searching when the location is known is limited to a descriptive analysis of 38 cases (Appendix B.1). The formula in Eq. 4 is not measured or used in any experiment.
W3. Several results that support the main claim are statistically weak or confounded.
-
The benefit side of the reversal is not confirmed in the replication. The reversal has two parts: guidance helps when the location is unknown and hurts when it is known. In the original study, the benefit under ordinary observation is clear (+21.00pp [+9.47, +32.47]; Table 9). In the new-seed study, however, it is +13.00pp [−0.31, +26.16], and the interval includes zero (Fig. 10). So the new data do not confirm that guidance helps when the location is unknown.
-
Fewer clean replications than claimed. The paper states that the reversal repeats across models. Of the three additional models, however, Mistral-7B does not show the reversal, and the Gemma experiment changes the model family, quantization, and inference backend at the same time. Only Mistral-24B is a clean replication.
-
Different experiments use different models. The controls showing that guidance content matters use Qwen2.5-7B and Llama-3.1-8B (Appendix A.1), the main reversal uses Qwen3-32B-AWQ, and the BrowserGym experiments use Ministral and Mistral models. The paper repeatedly notes that these protocols are separate and not pooled. As a result, the pieces of evidence cannot be combined into one argument. For example, it is not shown that the guidance content itself matters for the model used in the main study.
W4. The presentation is difficult to follow.
The paper introduces many overlapping terms (residual need, policy–state mismatch, information saturation, transport check) without clear operational definitions, and internal experiment labels appear throughout (“September 2026 control experiments,” “V4 follow-ups,” “M3 arms,” “P1 examine diagnostic,” “C2”).
Questions for authors
-
Please include the full prompts in the paper: the search-execute guidance, the matched control, and representative examples of the irrelevant, target-location, and destination facts. Figure 1 shows only a fragment of the control, and the guidance text is described only by its directive, token, and word counts (Appendix A.1).
-
Appendix A.1 indicates that the guidance consists of imperative directives. How does the reversal change if the same procedure is framed as advisory guidance (e.g., “agents often succeed by first searching systematically; consider whether this applies”) rather than as commands? This is different from the conditional wording tested in Sec. 4.4, and it would help distinguish whether the harm arises from the procedure itself or from the model treating imperative instructions as overriding the information in its context.