acceptodds
Withdrawn from ICLR 2027

WebSAPE: Structurally Aware and Progressively Experienced Web Agent

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Current web agents lack mechanisms to autonomously acquire structural knowledge of website architecture or to accumulate operational experience over time, and consequently regress to blind exploration on complex tasks. To address this, we present WebSAPE, a Structurally Aware and Progressively Experienced web-agent framework. WebSAPE maintains a centralized knowledge store consisting of (i) per-site textual topology guides synthesized from website exploration, summarizing page functions, reusable operations, and expected destinations, and (ii) an experience repository of operational lessons distilled from historical trajectories. A progressive learning pipeline is built to update the knowledge one-shot and incrementally. At runtime, both artifacts are jointly retrieved and injected into the agent's context to facilitate its execution. Additionally, we introduce BizArena, a high-fidelity benchmark for web agents in enterprise productivity environments, with 100 curated tasks spanning email, calendar, chat, project management, and cross-app workflows within self-hosted environments under deterministic, instrumented evaluation. Experiments indicate that our WebSAPE surpasses the state-of-the-art academic and industrial methods on the WebArena (+13.6 pp), WebArena-Lite (+10.3 pp), WebVoyager (+2.4 pp), and BizArena (+4.0 pp) benchmarks, demonstrating that integrating structural priors with progressively distilled experience yields markedly more capable web agents. Code and benchmark are publicly released at https://github.com/papersubmissionstore/WebSAPE.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Web agent, progressive learning, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9UgLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper introduces WebSAPE, a web-agent framework that combines structured website-topology guides with an experience repository that is progressively updated from prior trajectories. The framework retrieves these artifacts during execution to reduce unnecessary exploration and improve task completion. The paper also introduces BizArena, a deterministic benchmark for enterprise productivity workflows, and evaluates WebSAPE across several established web-agent benchmarks and the proposed enterprise setting.

Critical strengths

Strengths

  1. The method integrates complementary forms of knowledge that address practical agent limitations. Website topology provides structural guidance about possible navigation paths, while accumulated experience supplies operational lessons from previously executed tasks. Combining these artifacts gives the agent both a representation of where it can navigate and reusable guidance about how to execute tasks, which is a sensible design for complex, multi-step web workflows.

  2. The paper provides broad benchmark coverage and meaningful component ablations. WebSAPE is evaluated on WebArena, WebArena-Lite, WebVoyager, and BizArena, and the paper reports gains over several academic and industry systems. The ablations that remove experience and topology guides help quantify the contribution of the two knowledge sources rather than attributing the entire gain to the combined framework.

  3. BizArena adds a useful evaluation setting for enterprise productivity workflows. The benchmark includes email, calendar, chat, project management, and cross-application tasks in controlled environments. Its state isolation and evaluation of unintended side effects are useful design choices because enterprise tasks often require persistent state changes, where an apparently successful interaction may still corrupt or delete the wrong information.

Critical weaknesses

Weaknesses

  1. The methodological novelty relative to existing memory- and skill-based agents needs further clarification. The framework combines website-topology guides, distilled experience, retrieval, and hierarchical orchestration. While this integration is practically motivated, several of these components have clear precedents in earlier web-agent and memory-augmented systems. The paper should more clearly distinguish the proposed progressive learning mechanism from ordinary trajectory summarization, graph-based navigation guidance, and retrieval of previously successful skills. A more controlled comparison with closely related methods would strengthen the case for the paper's central algorithmic contribution.

  2. The effect of the progressive experience-learning mechanism is not fully separated from other system components. The framework also relies on a hierarchical orchestration setup and an underlying execution framework. Although the component ablations demonstrate the value of topology and experience, it is difficult to determine how much of the overall gain comes specifically from progressive experience refinement, how much comes from structural guidance, and how much comes from the broader agent scaffold. More controls that hold the orchestration and execution system fixed would improve the causal interpretation of the results.

  3. The enterprise benchmark is useful but still limited in scale and environmental diversity. BizArena contains 100 curated tasks in self-hosted, deterministic environments. This is a reasonable starting point for reproducible evaluation, but it provides limited evidence about generalization to unseen enterprise applications, larger workflows, or environments with concurrent changes and imperfect feedback. Further evaluation on larger task sets and unseen application configurations would help establish whether the observed improvements persist beyond the benchmark's specific scenarios.

Questions for authors
  1. Clarification The progressive experience-learning process updates the knowledge store based on preceding trajectories and evaluator feedback. Could the authors clarify the exact data protocol for the sequential evaluations: are the reported evaluation tasks used to update the knowledge store before subsequent tasks are evaluated, and what feedback is available for these updates? This is important for distinguishing online adaptation from generalization to an independently held-out task set and for ruling out potential task-order or evaluation leakage effects.

  2. Assessment-critical evidence WebSAPE combines topology guides, progressively distilled experience, and hierarchical orchestration. Could the authors clarify which experimental comparisons isolate the contribution of progressive experience learning independently of topology guidance and the underlying orchestration framework? In addition, are the main comparisons against alternative agent systems controlled for model, tool access, and inference budget? These details would materially affect the assessment of the method's independent algorithmic contribution versus gains arising from system integration.

Reviewer_hchLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduced WebSAPE, a new training-free web agent framework that used the topology guide and past experience to augment the agent's performance. Empirical results show that this framework surpassed the previous frameworks across several benchmarks. The authors also proposed a new benchmark to cover more agent tasks.

Critical strengths
  1. The empirical results show that WebSAPE is better than the current multiple SOTA frameworks.
Critical weaknesses
  1. The novelty is limited: two core ideas of this paper including using the topology and the the experience from past trajectories are both already proposed by previous works. This paper is more like a engineering work to develop a framework based on this proposed techniques.
  2. The experience are learned from the same enrivonment with the testing benchmark, which means that the comparison is unfair.
  3. The cross-validation experiment does not make sense. The paper did the cross-validation experiment on two benchmarks WebArena and WebArena-Lite. However, WA-Lite is almost a subset of WebArena, which means that they are in similar distributions.
  4. The progressively learning strategy is not verified by the experiments.
  5. The BizArena lacks a lot of details. And It's not clear what is the main benefit of this benchmark compared with previous works.

[1] Agent S:Saaket Agashe et al., “Agent S: An Open Agentic Framework that Uses Computers Like a Human,” ICLR 2025, arXiv:2410.08164 [2] ACE:Qizheng Zhang et al., “Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models,” ICLR 2026, arXiv:2510.04618

Questions for authors

See weaknesses.

Reviewer_8VcZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes WebSAPE, where a web agent adds two kinds of retrieved information to an Eko-based planner-executor. These include topology guides, which are site-wise guides using BSF, and an experience repository, where lessons are distilled from past trajectories by an LLM coach generator. The paper also introduces a new benchmark called BizArena. The benchmark has 100 tasks of synthesized email, chat, and project management apps. The authors reported experiments on WebArena, WebArena-Lite, WebVoyager, and BizArena, where WebSAPE outperformed baselines like Claude Code and Browser-Use.

Critical strengths
  • The core problem and motivation of the paper are very interesting. The authors identified some genuine gaps which they tried to address.
  • The method itself is simple, requires no fine-tuning, and can be used with any backbone.
  • The analysis of step-count and cost has been a very useful addition.
  • I also found the paper to be well-presented and a polished work, in general.
Critical weaknesses
  • The biggest concern is the empirical evidence of the paper. The authors have not reported whether these are single or multi-seed runs. Web agents have been very inconsistent across benchmarks (https://arxiv.org/abs/2608.18066), and the gains reported by the authors should be statistically significant.
  • The only identical backbone used is Opus-4.5. This has a few problems (a) we cannot generalize the results based on a single backbone. It could be a model-specific thing, (b) Opus-4.5 is a pretty outdated model at this point, (c) the authors didn't specify whether the thinking mode or thinking tokens were used. Did the LLMs have access to an external or scratchpad memory?
  • The authors also have not reported the observation space of the agents. This is very important for apples-to-apples comparison. An agent running on text only will not be able to compete with agents that receive both text and visual information. Benchmarks such as WebVoyager require vision to solve certain tasks. Since Opus-4.5 is the LLM backbone, I'm assuming that no vision component exists. In that case, maybe a captioning model can be used to add visual information to the blind model.
  • On the same note of not mentioning the observation space, the performance can also vary if the DOM or the accessibility tree is used. The authors have not discussed or ablated this.
  • The baselines ported seem weak. AgentOccam at 12.5% on Chat suggests a poor adaptation. Can the authors clarify how they have ported and share the code as well?
  • The authors reported multiple baselines without sharing the identical backbone. These cannot be compared directly. They should run some important baselines, such as Skill-based ones like SkillWeaver.
  • The zero-shot results of Opus-4.5 should be reported. Otherwise, we cannot determine how much of the gain is attributable to the harness enhancement.
  • The exploration component heavily resembles Go-Browse (cited in the paper, https://arxiv.org/abs/2506.03533) and AutoManual (https://arxiv.org/abs/2405.16247). The authors should discuss how the proposed exploration differs from the existing methods.
  • The cost comparison is incomplete. It excludes crawling, multi-epoch experience generation, and coach calls.
  • Without topology and experience, WebSAPE scores 55.5 on WebArena, below Claude Code and Browser-Use, and 65.0 on BizArena vs. 73.0 for Browser-Use. So the base scaffolding used by the authors underperforms pretty badly. The authors could try a better base scaffold and use topology+experience.
  • The abstract overstates the gains. The +13.6 pp and +10.3 pp figures compare an Opus 4.5 system against published numbers that mostly used older models. These gains come not only from methodological improvements but also from using a better backbone. The numbers against identical backbones should be reported.
  • The generalization evidence seems weak. WebArena tasks are instantiated from shared intent templates, so disjoint task splits still overlap at the template level. This likely explains why Vanilla WA to WA-Lite (69.1) beats the same-split result (67.9).
  • Web benchmarks are getting saturated, and there are several other benchmarks similar to BizArena, like AgentCompany (https://arxiv.org/abs/2412.14161), and OfficeBench (https://arxiv.org/abs/2407.19056). These benchmarks are like supersets of BizArena and cover most of the aspects that the authors proposed as differentiating factors. The authors should clearly state what existing gap in the benchmarking literature they have addressed.
Questions for authors
  • Can the authors run multiple seeds for the reported benchmarks? And also report the standard error across seeds.
  • I'd also appreciate it if the authors could report the results of other LLMs apart from Claude Opus 4.5 as the backbone. I would recommend at least two other LLMs from different families.
  • I'm also curious how open weight models would fare in the current pipeline. The authors should report the results of at least one open-weight model.
  • Can the authors discuss the motivation behind BizArena, and why such a benchmark is needed in the saturated space of web benchmarks?
  • It would be good to include modern LLMs as Opus 4.5 has been quite outdated at this point. GPT-6 could be a good option.
  • Do the topology guides and experience improve the current baselines, like Browser-Use or Claude Code, when added to them?
  • As the authors only ran four baselines with an identical backbone, it would be good to try other baselines, such as skill-based ones like SkillWeaver, and RL pipelines like WebRL.
  • It would be good to report the zero-shot performance using a vanilla harness as the floor, and fine-tuned performance as the ceiling (if the authors decide to add open-weight models).
  • Can the authors specify what the observation space is? Is it accessibility tree, DOM, screenshots, a set of marks, or a combination of these? Other papers have reported and ablated the observation space [VisualWebArena (https://arxiv.org/abs/2401.13649)]. I'm curious how the performance would vary across the observation spaces.
  • Most of the benchmarks are short-horizon. Can the authors test on long-horizon benchmarks, such as Odysseys?
  • Can the authors report which tasks were used to learn experience for each benchmark?
  • Can you please report results on a template- or site-disjoint split?
  • The authors mentioned results using Claude Code as a harness. Could they specify which version was used (for reproducibility)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.