- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that long-horizon web agents fail for two structural reasons, both stemming from latest-observation conditioning , i.e. a prompt containing only the instruction, the current observation and the growing list of past action labels:
- Trajectory-side failure ("effective information distance"). Decision-relevant content is present in the prompt but distant from the prediction site or scattered across turns. A controlled probe (Fig. 1, built from real WebArena trajectories, prompt variants over 12 tasks, Qwen2.5-14B-Instruct, greedy) varies placement while holding the gold answer fixed: short+concentrated 100%, long+concentrated 88.9%, short+distributed 48.1%, long+distributed 11.1%, long+distributed+state block 100%. The reading is that accuracy is governed by layout, not context length.
- Environment-side failure ("marginalization trap"). Because the model learns a marginal and the prompt has no slot in which an unobserved environment variable can be marked unknown, the policy silently substitutes a training-time default. A second probe (Fig. 2, 20 real baseline failure points, 4 regimes 5 cases) compares the standard trace with the same trace plus an oracle slot naming the missing fact: 7/20 (35%) 18/20 (90%). The latent-fact control is 5/5 in both; the three episode-specific regimes (execution error, stale locator, unverified write) go 0/5$\to$5/5, 2/5$\to$5/5, 0/5$\to$3/5. Method — CAST (Coded Agent with STate). The agent loop is rewritten as and . is a bounded, typed record (13 slots in the reported instance: trajectory bookkeeping, environment flags, termination fields) advanced by deterministic, per-slot code that reads only the most recent interaction — no learned weights, no retrieval, no extra LM call. Three principles generate the rules: action effect (error / URL change / observation change / repeat counts), termination evidence (task mode + visible absence or completion markers via a fixed regex lexicon), and budget. is a fixed prompt contract that ties state values to behavior: a stop-content clause (rejecting placeholders, hedged stops and empty strings; requiring a concrete value when evidence is visible) and a forbidden-action clause (barring an action class after repeats with no observation change, and both members of a 3-cycle). The key argument for the contract is that under latest-observation conditioning the previous observation is gone, so the policy cannot establish "this just had no effect" from its own input — the contract is what converts state facts into constraints.
Critical strengths
- A cheap, implementable design principle with a real effect on open models, and a non-obvious insight about why state must be paired with a contract. The "facts that admit a deterministic extraction rule belong in code, facts that require linguistic judgment remain in the policy" split is a clean partition, and CAST implements it at zero marginal inference cost — no extra LM call, no retrieval index, no exploration corpus, no training. On gpt-oss-20b this lifts WebArena from 14.7% to 25.6%, a 74% relative improvement, while matching a retrieval-based dynamics-grounding method that needs a GPT-4.1 exploration corpus. The accompanying insight — that alone is inert because a policy may read and ignore it, so a fixed is required to convert state values into constraints, and that the forbidden-action clause is not computable from the latest-observation prompt because the previous observation has been dropped — is a genuinely useful conceptual contribution: it separates "inform the policy" from "constrain the policy" and shows the latter needs the former. Consequential because this is an actionable, low-cost recipe that others can adopt immediately, and it is the part of the paper most likely to survive.
- The failure-mode decomposition as a diagnostic instrument. Figure 6b decomposes residual failures into premature stop, repeat loop, drift, hallucinated stop, step truncation and other, and shows the mechanism-level shift between variants: removing the contract block brings premature stop / repeat loop / hallucinated stop back, whereas removing state slots trades successes for step-truncated runs. This is more informative than a scalar success rate and gives the paper a defensible causal story for which component fixes which failure. Consequential because it is the only place in the paper where the claim "component X addresses failure Y" is tested rather than asserted, and it is the template a stronger version of this paper should follow on the full benchmark with proper statistics.
Critical weaknesses
- Every quantitative claim is under-powered, and the central claim fails precisely where it matters most — the strongest backbone. There is no seed variance, no confidence interval and no significance test anywhere in the paper. The three perturbation regimes use each (one task pp, so "+25.0" is six tasks); the ablation uses (one task pp, so every component contribution in the ladder — , , , — is exactly one task); the motivating probes use 27 prompt variants and 20 cases. Most consequentially, the three main-table results are not comparable in strength: the gain is monotonically decreasing in backbone capability (+10.9 pp on gpt-oss-20b, +4.6 pp on Qwen2.5-14B, +0.6 pp on gpt-5.4), and the gpt-5.4 result is a single run whose net gain is within any plausible noise band. Decomposing it by site makes the problem sharper: on gpt-5.4 the +0.6 pp average is the sum of +17.0 pp on Reddit (, tasks) and −11.7 pp on GitLab (, tasks), with Multi also negative (−4.1). GitLab is exactly the site that supplies +19.5 pp for the gpt-oss-20b headline (+3.9 pp on Qwen), i.e. the same site flips sign across backbones. Combined with the fact that Reddit and GitLab contribute roughly 75% of the gpt-oss-20b gain, the paper's claim that the effect "is spread across the six sites … targets no site-specific feature" and is "tied to the conditioning rather than to a particular model" is not supported. Consequential because the abstract's "improves success rate across backbones" and Section 1's "consistent gains across backbone scales" are the paper's headline claims, and the honest reading of Table 1 is "CAST substantially helps weak/open backbones and does nothing measurable for a frontier model, with a large site-level regression." Either the framing must change to a capability-dependent claim, or paired multi-seed runs with per-task outcomes (McNemar) and intervals must be supplied.
- The evidence that the two diagnosed bottlenecks are the binding constraints is thin and partly circular, and the stress subsets are selected on the outcome. Figure 2 — the sole empirical support for the "marginalization trap" — hands the model the ground-truth missing variable in an oracle slot, so "the correct action becomes selectable once the missing fact is named" is close to tautological; the control regime (latent fact) shows no effect at all (5/5 both), and the unverified-write regime recovers only to 3/5 even with the oracle, which undercuts the claim that a representational locus is "the fundamental remedy". , one backbone, greedy decoding. Figure 1 is the paper's most quotable number (11.1% at LD) and it rests on 27 prompt variants drawn from 12 tasks, of which 3 are "calibration variants of a single task" that were "added so that LD is non-zero"; since LD scores exactly 3/27, the natural inference is that LD is on the real items, i.e. the headline collapse is carried by three hand-inserted rows of one task. The perturbation study compounds this: Section 6.2 states the subsets were constructed to isolate "regime[s] where the GenericAgent baseline is known to drift", and the failures themselves are injected by the authors, so the +16.7 to +25.0 pp gains measure whether CAST handles the failure modes it was built for, not robustness in general — and the AI-use statement says LLM assistants "selected fixed WEBARENA task subsets used during development", so even the selection is not fully auditable. Consequential because Section 3 is the paper's motivation: if these probes are illustrative rather than probative, the paper is an empirical systems paper with a weak baseline, not a diagnosis of why agents fail.
Questions for authors
- Statistics. How many independent runs per condition are reported? Can you provide paired per-task outcomes and intervals for Table 1, the three 24-task subsets and the 53-task ablation? Is the gpt-5.4 average gain (+0.6 pp) distinguishable from zero, and what explains the −11.7 pp GitLab regression on gpt-5.4 given that GitLab is the site driving the gpt-oss-20b gain? (Clarification.)
- Baselines. Can you run at least one memory/reflection baseline (e.g. an AgentOccam-style observation-formatting baseline, or a reflection/AWM-style memory) in your own harness on the same backbones? Does CAST still add value over those, and does a contract-only prompt (stop-content + loop-breaking clauses, no state slots) capture most of the gain? (Clarification — this determines whether the contribution is the state or the prompt.)
3 comments
Response to Reviewer me3v (2/3): w2
w2: The evidence that the two diagnosed bottlenecks are the binding constraints is thin and partly circular. Figure 2 hands the model the missing fact through an oracle slot. Figure 1's LD result is carried by three calibration rows. The stress subsets are selected where the baseline drifts, with failures injected by the authors, and LLM-selected subsets are not fully auditable. If the probes are illustrative, the paper is an empirical systems paper, not a diagnosis.
R w2: Thank you for this careful analysis. We agree that the two probes in Section 3 are too small and too constrained to establish that the diagnosed bottlenecks are the binding constraints by themselves, and that Figure 4 does not measure robustness to general environment shifts. We accept your characterization of the paper as an empirical systems study of a low-cost component, with Section 3 serving as motivation. Each item is addressed below.
Figure 1. Your inference is correct: LD answers 0 of the 24 non-calibration items, and its 3 correct responses are the three calibration variants, which Appendix B.1 notes were added so that LD is non-zero. The conditions also differ in more than placement: SC, LC, and LD+S each state the resolved answer, SD splits it among distractors, and LD is a truncated failed history that lacks sufficient evidence. Figure 1 is therefore a diagnostic upper-bound probe of how a given answer is used under different presentations, not evidence that spatial layout alone causes the drop. A new comparison that changes only position (31 tasks, same evidence verbatim, no answer field) gives 23/31 vs. 25/31 for raw page evidence (4 improvements, 2 regressions) and 25/31 vs. 26/31 for distilled facts (2 and 1), with task-family bootstrap 95% CIs of [-14.29, +20.93] and [-8.34, +13.90] percentage points (both including zero). A full example is in our response to Reviewer 9f4i (1/2).
Figure 2. We agree on its scope: 20 cases, one backbone, and an oracle slot, which Appendix B.2 notes cannot show that a deployed pipeline can populate the slot at run time. Two clarifications: the latent-fact cases are at ceiling in both conditions (5/5), so the change comes from the other 15 cases (2/15 to 13/15); and the unverified-write regime reaches only 3/5 even with the oracle, which shows that naming the missing fact is not always sufficient without a behavioral constraint. We will remove "the fundamental remedy" and state the marginalization account as a motivating hypothesis rather than a proven bottleneck.
Stress subsets (Figure 4). One factual correction: no failure was injected. The three groups are subsets of unmodified tasks, selected by fixed deterministic rules from existing runs. Appendix C described a planned intervention design that was not run for Figure 4, which was our error; the exact rules and all 69 task IDs are in our response to Reviewer 9f4i (2/2). Your point about outcome-based selection applies directly to Group 2, which selects on the baseline's own rollout outcome (terminal status ERROR/TRUNCATED or at least 25 steps), so its gain is biased upward. Groups 1 and 3 are selected on task properties only (instruction value-lookup keywords; program_html evaluator type). In the original runs, the tasks solved only by the baseline vs. only by CAST are 1 vs. 6, 1 vs. 5, and 0 vs. 6 (two-sided exact McNemar p = 0.125, 0.219, and 0.031, unadjusted; Fisher exact p = 0.23, 0.19, 0.11). Figure 4 therefore measures performance on targeted task subsets rather than robustness to perturbations, and we will rename and describe the groups accordingly.
Auditability. In the AI-use statement, coding assistants implemented and ran the deterministic filtering scripts under our direction; the resulting task-ID lists are fixed files. We will include the selection rules and task IDs for the Figure 4 groups, the 53-task ablation set, and the probe items in Appendix C and in the anonymous repository.
Summary and planned revisions.
- Role of the probes and subsets. We will present the probes in Section 3 as motivating diagnostics rather than standalone proofs of the two bottlenecks, note that Group 2 of Figure 4 is selected on baseline outcomes, and frame the paper as an empirical study of a low-cost runtime state-and-contract component.
- The base agent. Our base agent follows the implementation of Chen et al. (2025), who report 30.0% for GPT-4.1 in the same setting; this gives context for why our absolute numbers sit below BrowserGym leaderboard systems that use different prompts or search, though it does not fully explain the level of our GPT-5.4 score.
We will present Section 3 as motivation, remove "the fundamental remedy" and "solely due to spatial layout", and rewrite Section 6.2 and Appendix C to match the actual selection rules.
Response to Reviewer me3v (3/3): q2
q2: Can you run at least one memory or reflection baseline in your own harness on the same backbones? Does CAST still add value over it, and does a contract-only prompt (stop-content and loop-breaking clauses, no state slots) capture most of the gain? This determines whether the contribution is the state or the prompt.
R q2: Thank you for this question, which goes to the core of how state and contract interact. Full 812-task evaluations of four additional variants in our harness (model-written memory, raw observation diff, state-only, and contract-only) are currently in progress and not yet complete in time for this response, so we do not claim a completed full-benchmark quantitative split between them here. Below we state how the four variants are defined, what our existing evidence and the architecture of the prompt establish, and what remains an open empirical question until those runs finish.
1. Why a contract-only prompt cannot replicate the loop-breaking condition in our harness. In our latest-observation conditioning setup, the policy prompt at step contains the task instruction, the current page observation , and the list of past action strings ; the previous observation is dropped.
- The stop-content clause of the contract (forbidding placeholder strings such as
'Done'or'Upvoted'and requiring a literal value) is a pure prompt instruction and does not depend on state slots. - By contrast, the forbidden-action clause ("if
same_action_repeat_count >= 3with no observation change, ortwo_action_cycle_count >= 3with no observation change, that action class is forbidden") depends on whether consecutive observations changed () and whether the last action returned an error. Because is absent from the baseline prompt, a contract-only prompt with no state slots can only give generic advice ("do not repeat an action that had no effect"), leaving the policy unable to verify from its current input whether the previous action changed the page. Conversely, a state-only prompt supplies those facts without explicitly forbidding the next repeat.
2. In-harness variants under evaluation.
- Model-written memory. The policy may emit an optional
<memory>...</memory>note in the same response as its action, carried into subsequent prompts without a separate memory-generation call. - Observation diff. The policy receives deterministic line additions and deletions between consecutive raw accessibility-tree observations, computed before normalization or prompt truncation.
- State only. The state block is added and the contract is removed.
- Contract only. A rewritten, standalone instruction set expressed in terms of visible evidence and action history, without the CAST runtime-state block or the executable stop guard.
3. Other baselines and the 53-task ablation.
- AgentOccam, Reflexion, and AWM. AgentOccam restructures the current accessibility tree and action space without extra calls, addressing single-step observation formatting rather than cross-step state tracking. Reflexion-style methods retry across multiple episodes after failure, and AWM induces workflows across tasks; both operate across episodes rather than within a single-attempt rollout with no separate state-maintenance call. We will add a comparison table and discussion of AgentOccam, AWM, ABBEL, PABU, and VLAA-GUI in Section 2.
- The 53-task ablation and full-benchmark outcome shifts. In Section 6.3, the cumulative ladder on the 53-task subset has a total spread of 3 tasks (30/53 to 33/53), which is too small a sample to statistically resolve the individual contribution of each component. At full-benchmark scale, Figure 5 shows where outcomes move on gpt-oss-20b: episodes that end with a stop and no reward fall from 578 to 402 of 812, while truncated episodes rise from 124 to 201 and successes from 110 to 204. This does not separate state from contract, which the full-benchmark variants in progress will test.
Thank you again for the detailed and constructive review.
Response to Reviewer me3v (1/3): w1 / q1
Thank you very much for the thorough review. We are grateful for your assessment that the split between code-computed facts and language judgment is a clean partition, and that pairing a state that informs with a contract that constrains is a useful insight. Your reading of Table 1 is the right one, and we will adopt it in the revised paper.
w1 / q1: Every quantitative claim is under-powered: no seed variance, intervals, or tests, and small subsets. The gain decreases with backbone capability. On GPT-5.4 the +0.6 average is the sum of +17.0 points on Reddit and -11.7 on GitLab, the site that drives the gpt-oss-20b gain, and Reddit and GitLab supply about 75% of that gain. How many independent runs are reported, can you provide paired per-task outcomes and intervals, is the GPT-5.4 gain distinguishable from zero, and what explains the GitLab regression?
R w1 / q1: You are right on each point, and we should have reported the site-level analysis ourselves. In Table 1, we report one evaluation per condition, without confidence intervals or significance tests. The GPT-5.4 gain is not distinguishable from zero. We will remove "improves success rate across backbones" and "consistent gains across backbone scales", and the statements that the effect is "spread across the six sites" and "tied to the conditioning rather than to a particular model". The revised paper states the capability-dependent claim you describe: CAST substantially helps smaller open-weight backbones and gives no measurable gain on GPT-5.4, where it also causes a site-level regression on GitLab.
Runs and statistics in Table 1. Table 1 reports one evaluation per condition. As noted in Table 1, for Qwen2.5-14B-Instruct the baseline (17.0) and GTTA (21.0) entries are the numbers reported by Chen et al. (2025), so that row is not a same-environment comparison, and we will mark it more prominently. In Table 1, gpt-oss-20b goes from 119 to 208 of 812 tasks (+10.9 points, unpaired 95% interval [+7.1, +14.8]) and GPT-5.4 goes from 285 to 290 (+0.6 points, [-4.0, +5.3], p = 0.80).
Paired per-task outcomes. In our recovered per-task paired records for gpt-oss-20b on all 812 tasks (108/812 for the baseline vs. 204/812 for CAST, from an earlier grading of the same runs), 36 tasks are solved only by the baseline and 132 only by CAST (+96 tasks / +11.8 points; task-family cluster-bootstrap 95% interval [+7.45, +16.56], two-sided exact McNemar p < 1e-11). Multi-seed variance across repeated runs is not yet measured and will be stated as a limitation.
GPT-5.4 and the GitLab regression. The regression in Table 1 matches your calculation: GitLab goes from 86 to 65 of 180 tasks (-21 tasks; unpaired p = 0.025), next to a rise from 33 to 51 of 106 on Reddit (where the baseline run experienced rate-limiting errors). One plausible hypothesis is that on a stronger backbone, CAST's repeat-action prohibition and stop guard over-constrain valid exploratory steps or retries on multi-step GitLab workflows; however, because per-task logs for our GPT-5.4 run are not available and our endpoint access for re-running GPT-5.4 is currently unavailable, we cannot provide a verified per-task attribution of the 21 lost GitLab tasks between the state and the contract. We will report the GitLab regression prominently in the main text as a negative result on a frontier backbone.
Site concentration. On gpt-oss-20b, the per-site changes in Table 1 are GitLab +35, Reddit +32, Shopping Admin +18, Shopping +4, Map +1, and Multi-site -1 tasks. GitLab and Reddit supply 67 of the 89 additional tasks (75%) while holding 35% of the benchmark, and GitLab changes sign between backbones. This contradicts our earlier statement that the effect is spread evenly across sites, which we will remove.
Summary and planned revisions. Our results support a large, statistically significant paired gain on gpt-oss-20b concentrated on GitLab and Reddit, alongside no measurable overall difference on GPT-5.4 and a site-level regression on GitLab. We will revise the abstract, Section 1, Section 6.1, and Section 7 to state this capability-dependent claim, add the confidence intervals and paired statistics to Table 1, mark the Qwen2.5-14B-Instruct row as taken from prior work, and report the site-level breakdown and the GPT-5.4 GitLab regression in the main text.