- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SAGE, a web-agent architecture that replaces step-by-step reasoning with programmatic execution using reusable knowledge of the application. SAGE crawls a web application to build a reusable state-machine graph describing interface states, operations, and transitions. It then generates a Python program for each task, compiles abstract operations into browser actions, and executes it with minimal further reasoning. Evaluated on WebArena, it reports >90% accuracy with lower latency and cost, showing the effectiveness of the proposed approach.
Critical strengths
- The proposed approach provides an interesting method of storing application-level memory and generating one-time program for task execution, which provides insights on effective and efficient web agents.
- The paper reports impressively high results on WebArena with relatively low costs.
Critical weaknesses
- The experiments are constrained to WebArena, and it is unclear how the proposed method can generalize to other web agent benchmarks, especially the live web ones interacting with the real Internet (for example, Online-Mind2Web).
- It is unclear what types of the tasks the proposed method can cover. It seems that it needs to fully understand the environment for building the state memory, but for real-world applications, this might not always be fesible. React-styled online exploration might still be necessary for many of these tasks.
Questions for authors
- For target websites where one cannot effectively crawl all the pages and build the state memory, how will the proposed approach perform?
- It seems that there is a lack of comparisons to approaches that specifically use enhanced agent memory methods.
- According to 356, evaluation is carried out in a single execution attempt without online repair; then will there be execution errors and how to deal with runtime errors like this?
Response to Reviewer nNDZ
We thank the reviewer for the feedback and questions. The main changes in the paper (marked in blue) are:
Target use case. SAGE targets agents that use the same applications repeatedly, such as an organization's internal tools or a user's frequently visited sites. In this setting, the one-time cost of building the SMG is amortized over many tasks (Table 2). In settings with one-off tasks across many different websites, a reactive agent that explores each site on the fly is a better fit.
Task coverage. We agree that some tasks are more suited for reactive exploration, and SAGE uses it as a fallback: when execution fails, the Patcher diagnoses and repairs the plan step by step (Section 4.3). In our evaluation, the main class of tasks that one-shot planning handles poorly is search tasks, where search terms must be chosen before the results are observed (∼22 failures, Appendix D.2). Tasks over structured interfaces, such as forms, reports, and multi-page navigation, are where planning over the SMG helps most (Section 5.2, Appendix D.1).
Q1: For target websites where one cannot effectively crawl all the pages and build the state memory, how will the proposed approach perform?
SAGE does not need to visit every page to build its state memory. The SMG models interface templates, not individual pages: on a shopping site, every product's detail page shares the same layout, so all of them map to one state. The graph size therefore grows with the number of distinct templates rather than the amount of content (Appendix A.1). Discovering the pages of a website is a well-researched problem that search engines solve at web scale, and SAGE could borrow their techniques.
When the crawl is less thorough, the SMG is less complete and performance drops. On GitLab, without warm-up, our original crawler (\29.5)yields55.0130), yields 84.4 (Table 2a, Appendix D.5). When coverage is incomplete, the Patcher repairs the plan at runtime and records the repair in the SMG, so coverage improves with use. When available, developers could also derive the graph from source code and documentation rather than relying solely on black-box crawling.
Q2: It seems that there is a lack of comparisons to approaches that specifically use enhanced agent memory methods.
We have added Agent Workflow Memory (AWM) as a memory-based baseline. AWM induces reusable workflows from its own successful trajectories and adds them to its prompt for later tasks. We run it in its online setting with the same model (Claude Opus 4.6) on the same 655 tasks: after each task, an LLM judge decides whether the trajectory succeeded, and AWM induces workflows from the successful ones. Because AWM builds its memory from the evaluation tasks themselves, it receives more warm-up than SAGE. By the end of each domain, its memory holds 11 to 24 workflows (Appendix D.3). Even so, AWM reaches 58.6% overall success, 32.6 points below SAGE (Table 1), and is 6.1× slower and 10× more expensive per task (Figure 4).
Q3: According to 356, evaluation is carried out in a single execution attempt without online repair; then will there be execution errors and how to deal with runtime errors like this?
With the Patcher disabled, an action that fails at runtime causes the task to fail. In our experiments, none of SAGE's failures were caused by broken selectors or SMG structural issues after the online refinement (Appendix D.2). We disable the Patcher during evaluation so that each result reflects a single pass of the synthesized program over a fixed SMG, with no repair or learning at test time. This is a conservative setting that measures the quality of the SMG and the plan.