acceptodds
Under review as a conference paper at ICLR 2027

WebChallenger: A Reliable and Efficient Generalist Web Agent

14 views

Abstract

Autonomous web navigation remains challenging for LLM agents, and the strongest generalist systems rely on proprietary reasoning models whose inference cost is prohibitive for the repetitive tasks where such agents would be most useful. We argue this gap stems not from insufficient model capability but from agent architectures that fail to replicate three human cognitive advantages: selective attention to relevant page regions, persistent memory of website structure, and procedural fluency with common interaction patterns. We introduce WebChallenger, a web agent framework that addresses each gap through architecture design rather than model scale, built around PageMem: a structured page representation deterministically constructed from the DOM that exposes each page as a hierarchy of semantic sections with short summaries. On this substrate we build three mechanisms that mirror these advantages: an observation pipeline that reads only task-relevant sections, a memory built by a one-time offline crawl of each website, and workflows that execute multi-step interactions as single actions. Because all three operate over PageMem, the framework generalizes across websites without site-specific adapters. Using off-the-shelf open-weight models without fine-tuning, our system achieves 56.3% on WebArena, 48.7% on VisualWebArena, 51.0% on Online-Mind2Web, and 70.9% on WorkArena, approaching frontier proprietary systems at a fraction of the cost. With the model held fixed in each comparison, WebChallenger scores 39 points higher on WebArena-lite than the standard BrowserGym harness and 14 points higher than the top-ranked WebArena harness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Web agents, large language models, agent architectures, structured observations, agent memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fwZGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WebChallenger is a browser-agent harness built around PageMem, a shared representation for selective observation, website memory, and compound actions. PageMem divides the DOM according to fixed rules and attaches model-generated summaries to the sections. The agent selects sections, extracts information for the current task, and uses reusable workflows for interactions such as dropdowns and forms. Website memory comes from task-independent offline exploration.

The paper evaluates open-weight models without fine-tuning on WebArena, VisualWebArena, Online-Mind2Web, and WorkArena. It includes matched-backbone harness comparisons and component ablations. A separate frontier-model version, WebChallenger+, reports lower estimated inference costs than Claude Code and Codex, with similar observed success on small task subsets.

Critical strengths
  1. PageMem gives observation, memory, and action execution a common interface. The agent can work with page sections and reusable interaction workflows without site-specific action code. The results span simulated websites, live websites, and enterprise interfaces, using off-the-shelf open-weight models. This breadth makes the approach more convincing than a result on one environment.

  2. The matched-backbone comparisons show what changing the harness can accomplish without changing the controller model. On WebArena-lite, WebChallenger reaches 58.8% success with GLM-4-32B, versus 19.4% for GenericAgent. It also improves on WebTactix with two backbones. Removing the observation pipeline lowers success by 17.0 percentage points, and Appendix E.4 includes controls for history format and separate components of website memory. These are useful checks on where the gain comes from.

Critical weaknesses
  1. I find the performance comparisons difficult to interpret as controlled comparisons of the harnesses. Appendix B.2 gives WebTactix 15 reasoning rounds, up to four parallel branches, and a 10-row table limit; WebChallenger gets 30 agent steps and a 50-row table limit. A WebChallenger step may also involve multiple browser operations and model calls. The reported success difference is promising, but these settings do not give the systems comparable inference budgets or access to information.

The frontier comparison has a similar problem. WebChallenger+ has a cycle and time budget, while Claude Code and Codex are described as uncapped; their initial prompts and context-management policies also differ. Similar observed success on the 40-task and 45-task subsets, with differences within one estimated standard error, does not establish equivalent accuracy. The efficiency claim needs care as well. Table 3 reports 47.0M controller tokens with selective observation versus 36.0M without it, and Table 10 reports 154 controller calls per task. The paper does not yet account for the full cost or measure latency under a controlled setup.

  1. I do not understand the extraction-cache correctness argument given the prompt shown in the appendix. Appendix E.3 treats an unchanged section as sufficient for reusing its cached extraction, while acknowledging that history is absent from the cache key. But the ExtractDetails prompt in Appendix D.1.1 includes HISTORY, the task, and current-page context. Deterministic decoding only gives the same result for the same complete prompt. Keeping the section fixed while changing history does not meet that condition.

An earlier extraction could omit a fact that matters to a later subgoal. Updating the page summary would not recover information missing from the cached extraction. This does not show that the benchmark scores are wrong, but the stated cache argument does not establish the reliability of this reuse policy.

Questions for authors
  1. How sensitive are the WebTactix results to the table limits and execution budgets? Could you compare the methods on an existing backbone with the same table limit and inference budget, reporting model calls, browser operations, and paired task outcomes? For the frontier comparison, what are the paired-success uncertainty estimates, and how would common stopping limits affect the cost ratios?

  2. What is the total efficiency cost after including controller calls, VLM calls, and offline exploration? How is exploration cost amortized across tasks? A latency comparison under a fixed serving setup would help interpret the 154 controller calls per task and the higher controller-token count relative to the observation ablation.

  3. Was HISTORY included in the ExtractDetails prompt used for the experiments? If so, why is a cached extraction reused after history changes while the section content stays fixed? Comparing cached and fresh extractions under changing histories, and measuring any effect on task success, would address this concern.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fwZG, part 1/2

We thank the reviewer for the very detailed review, in particular for the questions on matched budgets against WebTactix, stopping limits and uncertainty in the frontier comparison, total cost, and the extraction cache. Each of them led to new experiments, which we report below together with the corresponding changes in the paper.

W1a / Q1. "Appendix B.2 gives WebTactix 15 reasoning rounds ... these settings do not give the systems comparable inference budgets or access to information."

We agree that the original comparison used each harness's own default limits. We have now run both harnesses across a range of limits, with the same backbone, API and reasoning effort, on WebArena-lite (165 tasks). Table 8 and a new success-against-cost plot (Figure 4, App. B.2) show every setting. Success rates:

Limits, same for both harnessesBackboneWebTactixWebChallenger
15 steps or rounds, 10 table rows (WebTactix's default)Gemini 3.8 Flash62.4%70.9%
15 steps or rounds, 10 table rows (WebTactix's default)GLM-5.3-Flash52.7%62.4%
30 steps or rounds, 50 table rows (WebChallenger's default)Gemini 3.8 Flash65.5%75.8%
30 steps or rounds, 50 table rows (WebChallenger's default)GLM-5.3-Flash57.0%66.7%

WebTactix's best setting (30 rounds, 50 rows, 8 branches) reaches 67.9% with Gemini, still below WebChallenger at its tightest limits (70.9%).

Budget. A WebTactix round and a WebChallenger step are different units, so we compare budgets by cost per task. On a fixed model and provider, cost counts every model call either harness makes, weighted by its length. WebChallenger costs 3.4–5.3× more per task because it reads each selected section in its own model call. To match cost, we ran a two-call version of WebChallenger: one call picks the sections, and a second reads their full content and chooses the action. (This version needs a long-context model. Its success is 2.5 points below the original pipeline with Gemini and 3.0 points above it with GLM. The original pipeline keeps every prompt short so that small local models can run it, which is why the paper uses it.)

BackboneBest WebTactix settingWebChallenger, two-call
GLM-5.3-Flash57.0% at USD 0.00769.7% at USD 0.008
Gemini 3.8 Flash67.9% at USD 0.06073.3% at USD 0.080

At about the same cost, WebChallenger is 12.7 points ahead with GLM and 5.4 points ahead with Gemini. All other results in the paper use the original pipeline.

Paired outcomes. Per-task WebTactix outcomes were not available, so App. B.2 gives unpaired 95% intervals. On 165 tasks each single comparison is about ±10 points wide, and WebChallenger is ahead in all four matched-limit comparisons, by 8.5 to 10.3 points.

AuthorsOpenReview ↗

Response to Reviewer fwZG, part 2/2

W1b / Q1 (frontier). "WebChallenger+ has a cycle and time budget, while Claude Code and Codex are described as uncapped ... how would common stopping limits affect the cost ratios?"

Stopping limit. You noted that WebChallenger+ stops at 100 tool calls or 60 minutes while Claude Code and Codex were uncapped. We applied the same limit to their logged runs and recomputed their cost. It falls by at most 5%, and WebChallenger+ stays 1.7–3.4× cheaper (Table 18, App. F.1).

Sample size. You noted that the 40- and 45-task subsets were small. We enlarged them to 80 WorkArena L3 tasks and 90 WebArena tasks and re-ran every matched pair. Δ is WebChallenger+ minus the baseline, in points:

BenchmarkBackbone (baseline)WC+BaselineΔPaired 95% CICost ratio (capped)
WebArenaGPT-6 Astra (Codex)82.280.0+2.2[−3.9, 8.5]2.2×
WebArenaGemini 3.8 Flash (Codex)77.876.7+1.1[−6.0, 8.2]3.3×
WebArenaFable 5.1 (Claude Code)82.284.4−2.2[−7.8, 3.1]2.5×
WorkArena L3GPT-6 Astra (Codex)98.898.80.0[−5.6, 5.6]2.9×
WorkArena L3Gemini 3.8 Flash (Codex)92.586.3+6.3[−3.2, 16.0]3.4×
WorkArena L3Fable 5.1 (Claude Code)95.091.3+3.8[−4.6, 12.4]1.7×

Every paired interval contains zero, so §4.3 now claims comparable success rather than equal success.

Prompts and context policies. We ran Claude Code and Codex as shipped and did not change their prompts or context management. The comparison therefore measures what a practitioner gets from each model's own harness, not harness parity (App. E.5).

Q2. "What is the total efficiency cost after including controller calls, VLM calls, and offline exploration? ... interpret the 154 controller calls per task and the higher controller-token count ..."

Total cost. WebChallenger+ uses one model that reads the screenshots itself and does no offline exploration, so its per-task cost in Figure 3 and Table 18 is its total cost. For the original WebChallenger, exploration is a one-time cost per site (also in the live setting, where it is paid on the first visit): 16M tokens on WebArena, about the same as 55 tasks, and 5% of all tokens after 1,000 tasks (App. B.3).

47.0M vs 36.0M tokens, and 154 calls per task. The paper did not explain this well, and App. E.4 now does (below Table 11). The 154 calls are short calls, so every prompt stays near 1.85k tokens, which is what lets a 32B model read large pages. Without selective observation the agent uses fewer tokens in total, but each prompt holds the whole page (8.8k tokens), it needs 1.5× as many steps, and success falls by 17.0 points. With frontier models, WebChallenger+ reads the same sections in far fewer calls and costs 2.3–2.5× less than the original pipeline on the same model.

W2 / Q3. "Was HISTORY included in the ExtractDetails prompt used for the experiments? If so, why is a cached extraction reused after history changes ..."

Yes. HISTORY was in the extraction prompt and is not part of the cache key. We ran the test you suggested: we disabled the cache, so every extraction is recomputed with the current history (WebArena-lite, GLM-4-32B).

SuccessModel callsTokens
With cache (paper)58.8% (97/165)25,42747.0M
No cache (fresh extraction)57.6% (95/165)25,72548.2M

The difference is two tasks, within one standard error (about 4 points). Cache hits are rare, because the agent must return to a section it has already read and find it byte-identical. Early extractions are also broad, because the prompt asks for every detail relevant to the whole task. The cache is a small efficiency gain and does not affect success in practice.

You are right that our deterministic-decoding argument did not hold: a cache hit can differ from a fresh call made with a later history. App. E.3 now drops that argument, describes cache reuse as a design choice, and reports this ablation.

We hope the new experiments and the revised text address the reviewer's concerns. We are happy to provide any further analysis during the discussion period.

AuthorsOpenReview ↗

Kind reminder to Reviewer fwZG

Dear Reviewer fwZG,

We thank you once again for your thoughtful review and feedback. As the discussion period ends in about 16 hours, we hope our responses and the revised PDF have addressed your concerns, in particular the comparison with WebTactix under varied limits and at matched cost (§4.2, App. B.2), and the frontier comparison on larger samples with paired confidence intervals and a common stopping limit (§4.3, App. F.1). We remain available in case you have any follow-up questions.

Best regards,
Paper Authors

Reviewer_cvVEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a web agent harness named WebChallenger. The underlying idea is to let the agent focus on relevant information in a data format more efficient than the DOM. The experiments demonstrate that the harness substantially improves the score compared to the fine-tuning-based approaches. The authors extend the approach to a multimodal setting, where they apply their harness to proprietary models, showing a promising cost improvement over the official harnesses.

Critical strengths
  • Substantial experiments, including a realistic multimodal setup with proprietary models, consistently show promising benchmark scores.
Critical weaknesses

This work's central focus boils down to (i) efficient page representation named PageMem (derived from the DOM) and (ii) information trimming (giving the agent only relevant information). I have the following concerns regarding this design, making it unclear what should be considered the contribution/originality of this work:

  • The method requires an offline crawl of the whole website (line 21). This is intuitively effective for a benchmarking setup where the same environment is used across multiple queries. However, it feels unclear how it scales to a realistic, dynamic web environment.
  • The high-level idea of "rewriting/summarizing something into a more efficient format", or “providing selected/summarized information to the agent for efficiency” is also a widely explored topic (not only outside of web agent domains as mentioned in Section 2, but also within the domain such as [1-3])

[1] https://arxiv.org/abs/2511.21398 [2] https://arxiv.org/abs/2510.03204 [3] https://arxiv.org/abs/2307.12856

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer cvVE

We thank the reviewer for two sharp questions: whether the offline crawl carries over to the live, changing web, and what PageMem adds beyond existing methods that show an agent less and more relevant content. We answer both below and have added a feature comparison with the three cited methods to the paper.

Q1. "The method requires an offline crawl of the whole website (line 21) ... unclear how it scales to a realistic, dynamic web environment."

You asked whether the offline crawl works outside a benchmark. It does not need to run offline: the agent can crawl a site the first time it reaches it. All section summaries of a page are independent calls made in parallel, so each level of pages costs about one model call of time, which is small next to the time of a task (we also clarify this in the paper now):

Time
Start on a new site (its first page's sections summarized in parallel)1.4 s
Explore the pages one click away, all in parallel1.4 s
For scale: mean time of one WebArena task (WebChallenger+, Fable 5.1, measured)76 s

Since each click by the agent anyways takes more than 1.4s, other than the first page delay - the algorithm is online without bottleneck.

  • The crawl is cheap. Crawling a new site to depth 2 (16 pages) costs under USD 0.01 with gpt-6-luna, once per site.
  • The crawl is optional. Without any memory, WebChallenger still reaches 50.9% on WebArena-lite (58.8% with it), 2.6× the standard BrowserGym harness (19.4%) on the same model.
  • It already runs on the live web. We crawled Online-Mind2Web's 136 live sites the same way (27M tokens in total), and PageMem re-summarizes a section when its content changes, so the memory does not stay frozen.

Revised: §4.2, App. B.3 ("Crawl time and online exploration"), §5.

Q2. "The high-level idea of 'rewriting/summarizing something into a more efficient format' ... is also a widely explored topic (... within the domain such as [1-3])"

We agree that showing an agent less, and more relevant, content is a well-studied idea, and §2 now discusses [1–3]. What we contribute is how the page summary is built and reused:

Training-freePicks sections by summary, then reads eachFree-form extractionPage structure reused across steps and tasksSame structure drives memory and actions
WebAgent [3]✗✗✗✗✗
FocusAgent [2]✓✗✗✗✗
Prune4Web [1]✗✗✗✗✗
WebChallenger✓✓✓✓✓

What this gives the agent:

  • Every prompt stays small. The mean prompt is 1.85k tokens; putting the whole accessibility tree in one prompt (8.8k tokens) loses 17.0 points (Table 4).
  • The agent reads free-form text, not markup. An extraction can combine or compute over what it reads, for example averaging several values; removing extraction loses 7.3 points (App. E.4).
  • One cached structure serves the whole agent. The same sections are reused across steps and tasks and also carry the website memory and compound actions, with no training.

What each method reads and how it selects content is in the new App. E.7 (Table 12), which §2 points to.

The trade-off is that this uses more model calls per page than one-pass filtering with the same model, in exchange for higher accuracy, especially with small models.

Revised: §2, "Observation Refinement".

We hope this clarifies how WebChallenger runs on the live web and what is new in PageMem. We are happy to answer any further questions.

Thank you for the rebuttal. I was unfortunately not convinced by the discussion.

Q1)

  • I do not think the proposed scheme initially involved a cheap agent that crawls and summarizes pages, while the new rebuttal discussion reads that the crawl is based on a cheap model like Luna, which never appeared in the first draft.
  • Crawl being optional feels like an overly benchmark-focused statement. The content of the real website can change, so I do not believe this is true in many practical applications.
  • It’s true that Online-Mind2Web is “live” in a sense that it uses a real website at the time the user evaluates their model on it. However, what I meant by “dynamic” is a setup where the website can update e.g., per query, which probably requires another crawl and summarization to update the content of PageMem to handle stale memory. I think the author's evaluation setup is not dynamic in this sense.

Overall, this comment rather made the exact sell/setup unclear. (I apologize in case I simply failed to interpret your discussion; please correct me in that case)

Q2)

The new comparison table is unsophisticated. Most of the columns feel overly specific for putting a “✓” for the proposal, where having “✗” does not necessarily seem to imply a drawback of the method.

AuthorsOpenReview ↗

Response to Reviewer cvVE

We thank the reviewer for the follow-up and for pointing out where our previous answer was unclear. It packed several points together, and we apologize for the confusion. To be clear up front: the method and the setup are the ones described in the submission. Our rebuttal only added cost figures and time measurements for running the crawl online (App. B.3). We hope the answers below clarify the setup and, if so, would be grateful if you would reconsider your score. If any questions remain, we would be happy to answer them before the discussion period closes in about 16 hours.

The setup in brief

  • At every step, WebChallenger divides the live page into sections and summarizes them (PageMem). The agent always reads and acts on the current page; nothing has to be built in advance.
  • Website memory (bookmarks, dropdown contents, cached section summaries) is built by crawling the site. In all reported experiments, the section summaries are written by the paper's vision-language model (Qwen2.5-VL-7B; Qwen3-VL-4B for VisualWebArena), and GLM-4-32B makes every decision (§4, App. A.1). Memory raises success from 50.9% to 58.8% on WebArena-lite (Table 4).

Q1.1. The cheap crawler. We did not add a cheap crawler agent. The crawl uses only the summarizer of the submitted setup (Qwen2.5-VL-7B; Qwen3-VL-4B for VisualWebArena). Since it needs no decision model, it can also run on a cheaper model than the agent, and we used gpt-6-luna only as a stand-in API model to put a price on that: about USD 7 for all 50M exploration tokens, and under USD 0.01 for a 16-page site (App. B.3). None of the reported results use it.

Q1.2. "Crawling being optional". By "crawl" we meant building the website memory, not reading the page, which happens live at every step. "Optional" was too strong: we meant that the agent does not depend on memory (50.9% without it), not that memory should be skipped. Since the crawl is cheap, it can simply run online, when the agent reaches a site or at the start of each query:

  • Money: no API cost with the paper's own summarizers, which run locally, and under USD 0.01 per 16-page site with gpt-6-luna.
  • Time: each page's sections are summarized by independent calls made in parallel, so each level of pages takes about one summary call. We measured about 2.2 s per call with Qwen2.5-VL-7B and 1.1 s with Qwen3-VL-4B, the summarizers of our experiments, on the same RTX 3090, and 1.4 s with gpt-6-luna as an API proxy (App. B.3). One agent step makes several sequential model calls and takes longer, so after the first page of a new site the crawl runs alongside the agent instead of making it wait.

Q1.3. Dynamic websites. We agree that none of our benchmarks change between queries. A site that does can be re-crawled at the start of every query, and this adds no latency to the task beyond the first page. With the per-call times above, the agent waits about one summary call (1 to 2 s) before its first action; as in Q1.2, the rest of the crawl then runs alongside the agent (App. B.3).

Q2. The comparison table. The columns are not features picked to favor our method. Each is a design decision that any method refining the agent's observation has to make, and each has a cost as well as a benefit:

ColumnDesign decision and its trade-off
Training-freeNo training for a new model or site; trained filters can be more accurate per call.
Picks sections by summary, then reads eachCoarse-to-fine reading gives more relevant content, but makes more calls than one-pass filtering.
Free-form extractionThe agent reads text written for the task instead of raw markup.
Page structure reused across steps and tasksSection summaries are kept instead of recomputed, at the cost of a crawl.
Same structure drives memory and actionsOne representation serves reading, memory, and multi-step actions.

Each decision has a measurable effect. On WebArena-lite with the same model (GLM-4-32B):

ColumnWhat we removeEffect
Picks sections by summary, then reads eachwhole page in one prompt−17.0 points; prompts grow from 1.85k to 8.8k tokens (Table 4)
Free-form extractionselected sections passed as raw content−7.3 points (App. E.4)
Page structure reused across steps and taskswebsite memory (summaries rebuilt for every task)−7.9 points (Table 4)
Same structure drives memory and actionscompound actions−9.7 points (Table 4)
Training-free(compared with fine-tuning)our harness recovers what fine-tuning bought in most cases (§4.1)

We will revise the table in the paper to state each column as a design decision with its trade-off, as above, rather than as check marks.

Reviewer_jpTyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a training-free web-agent harness built around hierarchical page decomposition, selective observation, offline website memory, and compound browser workflows. It evaluates multiple open-weight models across four benchmarks, performs matched-backbone harness comparisons and component ablations, and extends the decomposition approach to frontier models.

Critical strengths

Strong practical harness performance across multiple benchmarks. The main ablations are supplemented by more targeted experiments separating observation from history, memory components, extraction, and action interfaces.

Critical weaknesses

Full results Table 5 should probably be in main paper, and WebChallenger with frontier API should also be included to evaluate how harness impact with the progress of frontier models.

Harness vs fine-tune conculsion is too broad, without full abalation of fine-tune on SOTA model with and without customized harness, the consludsion is immature.

Questions for authors

What are end-to-end latency and amortized cost, including vision calls and exploration, across different numbers of tasks per website?

WIth self-improving methods to optmize agent harness, e.g. browser-use [1] able to create a harness to achieve 97% on Online-Mind2Web (though with test-on-train setting), and DarwinX [2] show big improvement from 43.5% to 93% on WebArena-Infiniti with sim2real setting, I would suggest comparison with an self-improved harness.

[1] https://browser-use.com/posts/online-mind2web-benchmark [2] DarwinX: Evolving Agent Harnesses Through Natural Selection

3 comments
AuthorsOpenReview ↗

Response to Reviewer jpTy, part 1/2

We thank the reviewer for the careful reading and the concrete suggestions: moving the full results into the main paper, narrowing the harness-versus-fine-tuning claim, reporting amortized cost and latency, and relating our work to self-improving harnesses. We answer each point below and mark the corresponding changes in the revised PDF.

Q1. "Full results Table 5 should probably be in main paper, and WebChallenger with frontier API should also be included ..."

We agree and have moved it. The full table is now Table 2 in §4.1. Results for WebChallenger with frontier models are in two places in the main paper:

  • Same harness, stronger model (Table 3). On WebArena-lite (a 165-task subset of WebArena), WebChallenger reaches 46.7% with GPT-4o-mini, 58.8% with GLM-4-32B and 68.5% with GPT-5.
  • Frontier vision-language models (§4.3, Figure 3). WebChallenger+, the same page decomposition adapted to frontier VLMs, matches the success of the harness each model ships with (Claude Code or Codex) at 1.7–3.4× lower cost.

So the harness keeps adding value as the model gets stronger. Table 2's caption now points to both places.

Q2. "Harness vs fine-tune conculsion is too broad, without full abalation of fine-tune on SOTA model ... the consludsion is immature."

We agree that the sentence was too broad, and we have replaced it. We do not claim that a harness beats fine-tuning in general.

We took each published fine-tuned small open-weight agent and ran its base model, unchanged, inside our harness. In most cases this matched or beat the fine-tuned agent. For example, Qwen3.5-9B goes from 33.9% (fine-tuned) to 58.3% (our harness) on VisualWebArena. Two fine-tuned agents still beat us, and the paper already says so: Mobile-Agent-v3.5 is trained end-to-end on GUI trajectories and is 3.6 points ahead, and Qwen2-VL-7B is too weak for our structured prompts.

The ablation you describe (fine-tuning a SOTA model, with and without our harness) is beyond our compute budget. We see the two approaches as complementary, and fine-tuning a model for this harness is a natural next step.

Revised text (§4.1): "For small open-weight models at a fixed inference setting, a better harness thus recovered more than published fine-tuning in most cases; we do not claim it replaces fine-tuning in general, and the two are complementary."

AuthorsOpenReview ↗

Response to Reviewer jpTy, part 2/2

Q3. "What are end-to-end latency and amortized cost, including vision calls and exploration, across different numbers of tasks per website?"

In summary, exploring a website is a one-time cost. For WebArena it equals the tokens of about 55 tasks, falls to 5% of total tokens after 1,000 tasks, and adds no latency to a task. With a small API model, like GPT 6 luna - exploring every site in our benchmarks would cost about USD 7 in total.

Amortized cost. We explore each website once before inference, and every later task reuses the resulting memory. The summaries written during exploration come from the vision model, so these token counts include the vision calls.

BenchmarkExploration tokens (one-time)
WebArena (also reused for VisualWebArena)16M
WorkArena7M
Online-Mind2Web (136 live sites)27M

A WebChallenger task uses about 285k tokens (WebArena-lite). The 16M-token WebArena crawl therefore equals about 55 tasks. Its share of total tokens falls as more tasks run:

Tasks run on the WebArena sitesCrawl share of all tokens
10036%
1,0005%
1,722 (all our WebArena and VisualWebArena tasks)3%

Per site the share is lower. Shopping, the largest site, took 2.8M tokens, which is about 10% of the total after 100 Shopping tasks. Exploration needs only a summarization model. With a small API model (gpt-6-luna) through a batch API, all 50M exploration tokens would cost about USD 7.

Latency.

  • Crawl. The crawl is cheap: crawling a new site to depth 2 (16 pages) costs under USD 0.01 with an API model, and its summary calls run in parallel. It runs once per website, whatever the task, so it adds no time to a task.
  • Per task. We measured wall-clock time for one same-backbone pair, both running Fable 5.1 through the same API in the same period. WebChallenger+ took a mean of 283 s per task against 539 s for Claude Code on WorkArena L3, and 76 s against 154 s on WebArena. This was not a controlled serving setup, so we report it only as an indication.

The paper compares harnesses on tokens and cost. On a fixed model, cost is proportional to tokens, while latency also depends on provider load and caching. We also say in §5 that the original pipeline makes many short sequential calls, which adds round trips.

Revised: §4.2, App. B.3 ("Amortized cost", "Crawl time and online exploration"), App. E.5.

Q4. "With self-improving methods to optmize agent harness, e.g. browser-use [1] ... I would suggest comparison with an self-improved harness."

Self-improved harnesses have shown large gains, as your two examples show, and we agree the comparison is worth making. A fair version needs a careful setup. The Browser Use result tunes the harness on Online-Mind2Web's own tasks, so part of the gain may come from seeing the test tasks. A clean comparison would run the search on one set of websites and test on different websites, to measure whether the improved harness transfers to new sites. It should also report the cost of the search itself, since every candidate harness is run on the training tasks, next to our single exploration pass per site. Setting this up properly takes longer than the rebuttal period allows.

The two approaches also fit together. A harness search edits the control loop, prompts and tools of an existing harness. WebChallenger is hand-designed and not tuned on any benchmark's tasks, so it is a natural starting point for such a search. Running a search from WebChallenger, and from a simple generic agent for comparison, is the experiment we plan to run next.

Revised: §2 now discusses both [1] and [2].

We hope these answers and the revised paper address the reviewer's concerns, and we are happy to answer any further questions.

AuthorsOpenReview ↗

Kind reminder to Reviewer jpTy

Dear Reviewer jpTy,

We thank you once again for your thoughtful review and feedback. As the discussion period ends in about 16 hours, we hope our responses and the revised PDF have addressed your concerns, in particular the full results table now in the main paper (Table 2, §4.1), the narrowed harness-versus-fine-tuning claim (end of §4.1), and the exploration cost and crawl time (§4.2, App. B.3). We remain available in case you have any follow-up questions.

Best,
Paper Authors

Comments on the paper

2 comments

We thank all three reviewers. Every change in the revised PDF is shown in blue. Section, table and appendix numbers below refer to the revised PDF. The main text now uses the one extra page.

#ChangeWhereFor
1Full results table moved into the main paperTable 2, §4.1jpTy
2Harness-vs-fine-tuning claim narrowedend of §4.1jpTy
3Amortized exploration cost, crawl time; exploration is a one-time step that can also run online on a site's first visit§3.3, §4.2, App. B.3, §5jpTy, cvVE, fwZG
4Related work: [1–3] and self-improving harnesses§2cvVE, jpTy
5WebTactix vs WebChallenger under varied limits and at matched cost§4.2, Table 8 (App. B.2)fwZG
6Frontier comparison on larger samples, paired confidence intervals, common stopping limit§4.3, Table 18 (App. F.1)fwZG
7Extraction cache: corrected argument and a no-cache ablationApp. E.3fwZG
8Measured wall-clock times we haveApp. E.5jpTy, fwZG
9Success-against-cost plot of every WebTactix and WebChallenger settingFigure 4 (App. B.2)fwZG
10Why the original pipeline makes 154 short calls per taskApp. E.4fwZG
11Feature comparison with WebAgent, FocusAgent and Prune4WebApp. E.7, Table 12cvVE
12Frontier claim stated as comparable (not equal) success at 1.7–3.4× lower cost throughoutAbstract, §1, §4.3, §6jpTy, fwZG
13Main-text description of the extraction cache corrected§3.4fwZG

Dear Reviewers, pls can you respond if your concerns were addressed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.