- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WebChallenger is a browser-agent harness built around PageMem, a shared representation for selective observation, website memory, and compound actions. PageMem divides the DOM according to fixed rules and attaches model-generated summaries to the sections. The agent selects sections, extracts information for the current task, and uses reusable workflows for interactions such as dropdowns and forms. Website memory comes from task-independent offline exploration.
The paper evaluates open-weight models without fine-tuning on WebArena, VisualWebArena, Online-Mind2Web, and WorkArena. It includes matched-backbone harness comparisons and component ablations. A separate frontier-model version, WebChallenger+, reports lower estimated inference costs than Claude Code and Codex, with similar observed success on small task subsets.
Critical strengths
-
PageMem gives observation, memory, and action execution a common interface. The agent can work with page sections and reusable interaction workflows without site-specific action code. The results span simulated websites, live websites, and enterprise interfaces, using off-the-shelf open-weight models. This breadth makes the approach more convincing than a result on one environment.
-
The matched-backbone comparisons show what changing the harness can accomplish without changing the controller model. On WebArena-lite, WebChallenger reaches 58.8% success with GLM-4-32B, versus 19.4% for GenericAgent. It also improves on WebTactix with two backbones. Removing the observation pipeline lowers success by 17.0 percentage points, and Appendix E.4 includes controls for history format and separate components of website memory. These are useful checks on where the gain comes from.
Critical weaknesses
- I find the performance comparisons difficult to interpret as controlled comparisons of the harnesses. Appendix B.2 gives WebTactix 15 reasoning rounds, up to four parallel branches, and a 10-row table limit; WebChallenger gets 30 agent steps and a 50-row table limit. A WebChallenger step may also involve multiple browser operations and model calls. The reported success difference is promising, but these settings do not give the systems comparable inference budgets or access to information.
The frontier comparison has a similar problem. WebChallenger+ has a cycle and time budget, while Claude Code and Codex are described as uncapped; their initial prompts and context-management policies also differ. Similar observed success on the 40-task and 45-task subsets, with differences within one estimated standard error, does not establish equivalent accuracy. The efficiency claim needs care as well. Table 3 reports 47.0M controller tokens with selective observation versus 36.0M without it, and Table 10 reports 154 controller calls per task. The paper does not yet account for the full cost or measure latency under a controlled setup.
- I do not understand the extraction-cache correctness argument given the prompt shown in the appendix. Appendix E.3 treats an unchanged section as sufficient for reusing its cached extraction, while acknowledging that history is absent from the cache key. But the ExtractDetails prompt in Appendix D.1.1 includes HISTORY, the task, and current-page context. Deterministic decoding only gives the same result for the same complete prompt. Keeping the section fixed while changing history does not meet that condition.
An earlier extraction could omit a fact that matters to a later subgoal. Updating the page summary would not recover information missing from the cached extraction. This does not show that the benchmark scores are wrong, but the stated cache argument does not establish the reliability of this reuse policy.
Questions for authors
-
How sensitive are the WebTactix results to the table limits and execution budgets? Could you compare the methods on an existing backbone with the same table limit and inference budget, reporting model calls, browser operations, and paired task outcomes? For the frontier comparison, what are the paired-success uncertainty estimates, and how would common stopping limits affect the cost ratios?
-
What is the total efficiency cost after including controller calls, VLM calls, and offline exploration? How is exploration cost amortized across tasks? A latency comparison under a fixed serving setup would help interpret the 154 controller calls per task and the higher controller-token count relative to the observation ablation.
-
Was HISTORY included in the ExtractDetails prompt used for the experiments? If so, why is a cached extraction reused after history changes while the section content stays fixed? Comparing cached and fresh extractions under changing histories, and measuring any effect on task success, would address this concern.
3 comments
Response to Reviewer fwZG, part 2/2
W1b / Q1 (frontier). "WebChallenger+ has a cycle and time budget, while Claude Code and Codex are described as uncapped ... how would common stopping limits affect the cost ratios?"
Stopping limit. You noted that WebChallenger+ stops at 100 tool calls or 60 minutes while Claude Code and Codex were uncapped. We applied the same limit to their logged runs and recomputed their cost. It falls by at most 5%, and WebChallenger+ stays 1.7–3.4× cheaper (Table 18, App. F.1).
Sample size. You noted that the 40- and 45-task subsets were small. We enlarged them to 80 WorkArena L3 tasks and 90 WebArena tasks and re-ran every matched pair. Δ is WebChallenger+ minus the baseline, in points:
| Benchmark | Backbone (baseline) | WC+ | Baseline | Δ | Paired 95% CI | Cost ratio (capped) |
|---|---|---|---|---|---|---|
| WebArena | GPT-6 Astra (Codex) | 82.2 | 80.0 | +2.2 | [−3.9, 8.5] | 2.2× |
| WebArena | Gemini 3.8 Flash (Codex) | 77.8 | 76.7 | +1.1 | [−6.0, 8.2] | 3.3× |
| WebArena | Fable 5.1 (Claude Code) | 82.2 | 84.4 | −2.2 | [−7.8, 3.1] | 2.5× |
| WorkArena L3 | GPT-6 Astra (Codex) | 98.8 | 98.8 | 0.0 | [−5.6, 5.6] | 2.9× |
| WorkArena L3 | Gemini 3.8 Flash (Codex) | 92.5 | 86.3 | +6.3 | [−3.2, 16.0] | 3.4× |
| WorkArena L3 | Fable 5.1 (Claude Code) | 95.0 | 91.3 | +3.8 | [−4.6, 12.4] | 1.7× |
Every paired interval contains zero, so §4.3 now claims comparable success rather than equal success.
Prompts and context policies. We ran Claude Code and Codex as shipped and did not change their prompts or context management. The comparison therefore measures what a practitioner gets from each model's own harness, not harness parity (App. E.5).
Q2. "What is the total efficiency cost after including controller calls, VLM calls, and offline exploration? ... interpret the 154 controller calls per task and the higher controller-token count ..."
Total cost. WebChallenger+ uses one model that reads the screenshots itself and does no offline exploration, so its per-task cost in Figure 3 and Table 18 is its total cost. For the original WebChallenger, exploration is a one-time cost per site (also in the live setting, where it is paid on the first visit): 16M tokens on WebArena, about the same as 55 tasks, and 5% of all tokens after 1,000 tasks (App. B.3).
47.0M vs 36.0M tokens, and 154 calls per task. The paper did not explain this well, and App. E.4 now does (below Table 11). The 154 calls are short calls, so every prompt stays near 1.85k tokens, which is what lets a 32B model read large pages. Without selective observation the agent uses fewer tokens in total, but each prompt holds the whole page (8.8k tokens), it needs 1.5× as many steps, and success falls by 17.0 points. With frontier models, WebChallenger+ reads the same sections in far fewer calls and costs 2.3–2.5× less than the original pipeline on the same model.
W2 / Q3. "Was HISTORY included in the ExtractDetails prompt used for the experiments? If so, why is a cached extraction reused after history changes ..."
Yes. HISTORY was in the extraction prompt and is not part of the cache key. We ran the test you suggested: we disabled the cache, so every extraction is recomputed with the current history (WebArena-lite, GLM-4-32B).
| Success | Model calls | Tokens | |
|---|---|---|---|
| With cache (paper) | 58.8% (97/165) | 25,427 | 47.0M |
| No cache (fresh extraction) | 57.6% (95/165) | 25,725 | 48.2M |
The difference is two tasks, within one standard error (about 4 points). Cache hits are rare, because the agent must return to a section it has already read and find it byte-identical. Early extractions are also broad, because the prompt asks for every detail relevant to the whole task. The cache is a small efficiency gain and does not affect success in practice.
You are right that our deterministic-decoding argument did not hold: a cache hit can differ from a fresh call made with a later history. App. E.3 now drops that argument, describes cache reuse as a design choice, and reports this ablation.
We hope the new experiments and the revised text address the reviewer's concerns. We are happy to provide any further analysis during the discussion period.
Kind reminder to Reviewer fwZG
Dear Reviewer fwZG,
We thank you once again for your thoughtful review and feedback. As the discussion period ends in about 16 hours, we hope our responses and the revised PDF have addressed your concerns, in particular the comparison with WebTactix under varied limits and at matched cost (§4.2, App. B.2), and the frontier comparison on larger samples with paired confidence intervals and a common stopping limit (§4.3, App. F.1). We remain available in case you have any follow-up questions.
Best regards,
Paper Authors
Response to Reviewer fwZG, part 1/2
We thank the reviewer for the very detailed review, in particular for the questions on matched budgets against WebTactix, stopping limits and uncertainty in the frontier comparison, total cost, and the extraction cache. Each of them led to new experiments, which we report below together with the corresponding changes in the paper.
We agree that the original comparison used each harness's own default limits. We have now run both harnesses across a range of limits, with the same backbone, API and reasoning effort, on WebArena-lite (165 tasks). Table 8 and a new success-against-cost plot (Figure 4, App. B.2) show every setting. Success rates:
WebTactix's best setting (30 rounds, 50 rows, 8 branches) reaches 67.9% with Gemini, still below WebChallenger at its tightest limits (70.9%).
Budget. A WebTactix round and a WebChallenger step are different units, so we compare budgets by cost per task. On a fixed model and provider, cost counts every model call either harness makes, weighted by its length. WebChallenger costs 3.4–5.3× more per task because it reads each selected section in its own model call. To match cost, we ran a two-call version of WebChallenger: one call picks the sections, and a second reads their full content and chooses the action. (This version needs a long-context model. Its success is 2.5 points below the original pipeline with Gemini and 3.0 points above it with GLM. The original pipeline keeps every prompt short so that small local models can run it, which is why the paper uses it.)
At about the same cost, WebChallenger is 12.7 points ahead with GLM and 5.4 points ahead with Gemini. All other results in the paper use the original pipeline.
Paired outcomes. Per-task WebTactix outcomes were not available, so App. B.2 gives unpaired 95% intervals. On 165 tasks each single comparison is about ±10 points wide, and WebChallenger is ahead in all four matched-limit comparisons, by 8.5 to 10.3 points.