- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes PACT, a pipeline that generates page-coupled tools for web agents following the WebMCP draft. A coding agent collects task-independent page states by BFS from a site’s homepage, discovers page-native capabilities, specifies each as a single-function tool with a page-state registration condition, implements it as a JavaScript callback or annotated HTML form, and a separate agent verifies semantics, scope, execution and selection; execution terminates at any full-page navigation so composition stays with the calling agent. Integration is treated as a design variable (full vs. lightweight page × full definitions vs. catalog). On 812 WebArena tasks, four agents × four models all improve in success with the default integration, and 14/16 configurations use fewer tokens. Additional studies cover a naive and an agent-optimised tool set, integration and screenshot ablations, Qwen-3.5 model sizes, fine-tuned web agents, WebArena Verified, and four self-built mock sites.
Critical strengths
- Clear motivation with design choices that follow from it. The Introduction states four concrete limitations of WALT-style learned tools (L1 trajectory-bound coverage, L2 bundled navigation/agentic steps, L3 state-independent exposure, L4 unexamined integration), and each PACT component answers one: task-free evidence-based generation (L1), the single-function boundary at page navigation (L2), the registration condition (L3), and the observation × exposure design space (L4).
- The improvement from PACT is consistent and robust. Experimental results show that the performance gains from PACT upon the base agents do not decrease in any of the 16 agent × model configurations on 812 WebArena tasks and rises by up to +23.4 pp, with tokens down in 14/16.
- The integration study is informative. In section 3.3, the authors turn tool–agent integration into a 2×2 design space (full vs. lightweight page × full tool definitions vs. catalog) and Section 4.1.3 measures it on ColorBrowserAgent with two models over the same 812 tasks. The finding that the same tool set yields different accuracy and cost depending on what the agent sees is interesting.
Critical weaknesses
-
Positioning and novelty are overstated and two directly relevant works are uncited. SkillWeaver (Zheng et al., 2025) synthesises skills as APIs from self-exploration with a test-and-debug stage; WebXSkill (Wang et al., 2026) already exposes skills by page via a URL-pattern graph (the substance of L3) and studies deployment modes with model-dependent preferences (the substance of L4). The genuinely new elements are the in-page WebMCP execution substrate and trajectory-free generation; the single-function boundary and state-conditioned exposure are refinements of existing ideas.
-
No quantitative comparison with any existing tool- or skill-extraction method. Every quantitative comparison is Original vs. PACT or PACT-variant vs. PACT-variant; the only reference to an existing method is the qualitative Table 8.
-
A single benchmark, evaluated under a modified grader. Only one established benchmark, WebArena, is used; there are no results on WebVoyager, Online-Mind2Web, or live sites. The two supplementary evaluations do not change this: WebArena Verified (Table 4) is the same 812 tasks with corrected evaluators, run for one agent–model pair, and the mock Chinese sites (A.12, Table 20) are generated by Kimi-K3 with 90 author-written tasks, again for one pair. Section 4 further replaces the grader’s LLM with Qwen3.7-max, so the reported numbers are not comparable with published WebArena results, making it impossible to verify the effectiveness of PACT relative to other methods.
Questions for authors
Please see weaknesses.
4 comments
Author response for Reviewer e8Hg - 2
“Section 4 further replaces the grader's LLM with Qwen3.7-max.”
Thank you for raising this concern. We would like to clarify our motivation for using a different grading model. Our primary goal is to evaluate the effectiveness of PACT under a consistent evaluation protocol across different agents and tool configurations.
The WebArena codebase specifies gpt-4-1106-preview as its LLM grader. However, OpenAI announced on April 22, 2026, that this model would be shut down on October 23, 2026, after our submission deadline but before the rebuttal period. Therefore, relying on this model would prevent us from maintaining the same grading configuration throughout the evaluation and additional rebuttal experiments. We consequently adopted Qwen3.7-Max for all LLM-based grading checks, ensuring that all agent and tool configurations are evaluated consistently.
Evaluator agreement. To assess the potential impact of this change, we conducted an additional evaluator-agreement analysis using GPT-4o and GPT-5.6-Sol, which are the replacements for gpt-4-1106-preview, according to OpenAI. The results are reported in Appendix A.14 (Table 22).
| Condition | GPT-4o Agreement | GPT-4o κ | GPT-5.6-Sol Agreement | GPT-5.6-Sol κ |
|---|---|---|---|---|
| Main experiment (16 agent-model pairs) | 98.7% | 0.800 | 99.0% | 0.844 |
| AgentOccam (Qwen-3.7-Plus) + Verified Naive | 98.5% | 0.764 | 99.0% | 0.840 |
| WALT agent + WALT tools | 97.7% | 0.618 | 97.9% | 0.636 |
| CUGA + WALT tools | 98.5% | 0.753 | 99.0% | 0.830 |
For the main experiments, GPT-4o and GPT-5.6-Sol achieve 98.7% and 99.0% agreement with Qwen3.7-Max, respectively, with Cohen's κ of 0.800 and 0.844. The additional evaluations also show high agreement across different tool configurations.
These results support the consistency of our evaluation under different grading models. Although changing the grader limits direct numerical comparison with previously published WebArena results, our conclusions are based on matched comparisons under the same evaluation protocol. We further address cross-method comparability by evaluating WALT and PACT tools using the same CUGA agent, reasoning model, tasks, and grader (Appendix A.15).
Thank you for the candid response and the additional experiments. The controlled WALT comparison under the same CUGA agent and the Online-Mind2Web evaluation on live sites address my main concerns about missing baselines and single-benchmark evaluation, and I am raising my score accordingly. Some concerns remain:
- Novelty. As the authors acknowledge, the contribution lies in the integrated pipeline rather than in its individual mechanisms. The main remaining distinction from prior skill-based approaches is registration based on live page conditions rather than URL patterns, which is a modest increment.
- WALT comparison. The +3.9 pp gain comes from a single agent–model pair and a single run, without a significance test. The token advantage holds only relative to CUGA + WALT tools; WALT with its own agent uses fewer tokens (146.5k vs. 206.5k). It is also unclear whether the integration settings were tuned equally for both tool sets, and tool-generation costs are not compared. A quantitative comparison with SkillWeaver is still missing.
- Efficiency on live sites. On Online-Mind2Web, PACT increases tokens by 15.4% and steps by 7.7%. This differs from the efficiency gains reported on WebArena and suggests that the cost benefits may not carry over to live websites.
I encourage the authors to include these analyses and caveats in the revision.
Response to Reviewer e8Hg
Thank you for the updated assessment and the higher score. We would like to share a few additional details:
“Novelty ... the contribution lies in the integrated pipeline ... Page-conditioned registration is a modest increment.”
We would like to clarify that PACT differs from prior skill-based approaches in its objective, generation process, and execution semantics, beyond tool registration.
While SkillWeaver and WebXSkill acquire reusable agent skills from exploration experiences or execution trajectories, PACT automatically transforms existing website functionality into WebMCP-compatible tools. It derives tools from page states, DOM structures, and application logic without requiring task-execution trajectories or prior agent-side experience, making generation independent of specific agents.
Moreover, PACT exposes bounded, page-native operations rather than multi-step agent behaviors, leaving navigation, reasoning, and tool composition to the calling agent. Each tool is independently verified and dynamically registered based on live page conditions.
PACT's primary contribution is thus a trajectory-independent framework for exposing website functionality through WebMCP, while separating website operations from agent-level decision-making.
“Significance test.”
We conducted a paired analysis of 812 WebArena tasks using CUGA with Qwen-3.7-Plus. PACT achieved 44.3% success versus 40.4% for WALT, solving 99 tasks missed by WALT, while WALT solved 67 missed by PACT. The paired improvement was 3.94 percentage points (95% paired-bootstrap CI: 0.86–7.02; two-sided exact McNemar p = 0.0159), indicating a statistically significant improvement.
“The token advantage holds only relative to CUGA + WALT tools; WALT with its own agent uses fewer tokens (146.5k vs. 206.5k).”
We thank the reviewer for this observation. WALT with its own agent uses fewer tokens (146.5k vs. 206.5k), but also achieves a lower success rate (37.2% vs. 44.3%).
PACT aims to generate agent-independent tools that generalize across Web agents, rather than optimize for a specific agent. We therefore compare PACT and WALT under the same CUGA agent to isolate tool effectiveness. In this controlled setting, PACT improves the success rate from 40.4% to 44.3% while reducing token consumption by 23.9%.
We additionally evaluated PACT tools with the WALT agent. PACT achieves 40.3% success, compared with 37.2% using WALT tools. Mean recorded agent token usage decreases from 146.5k to 125.2k per task. Table 23 reports the corresponding resource statistics.
| Condition | SR (%) | Tokens (all) | Tokens (succ.) | Output tokens (all) | Output tokens (succ.) | Cache hit % (all) | Cache hit % (succ.) |
|---|---|---|---|---|---|---|---|
| WALT agent + WALT tools | 37.2 (+8.4 pp) | 146.5k | 123.9k | 4.1k | 3.4k | 45.9 | 48.0 |
| WALT agent + PACT tools | 40.3 (+11.5 pp) | 125.2k | 95.6k | 4.0k | 3.0k | 40.6 | 38.9 |
“Equally tuned for both tool sets, and tool-generation costs.”
Neither integration was tuned on WebArena. Both followed CUGA configurations with Qwen-3.7-Plus. PACT tools were registered based on live page states, while WALT's released workflows and tool catalog were adapted to CUGA's interface. Both operated within CUGA's browser context, with CUGA handling planning and action selection.
Appendix A.17 reports 1,381.0 million tokens for generating and verifying the 184 PACT tools, including page-state collection and revision but excluding the task-independence audit. Since we used WALT's released workflows, its original construction cost is unavailable. Table 23 compares execution costs only, excluding tool construction.
“A quantitative comparison with SkillWeaver is still missing.”
While SkillWeaver's implementation is publicly available, its pre-generated WebArena skills do not appear to be released. Unlike WALT, whose tools can be directly adopted, evaluating SkillWeaver requires reproducing its skill generation.
This process is ongoing but has been delayed by slow API responses and the large number of required calls. We plan to include the comparison in the revised manuscript once completed.
“On Online-Mind2Web, higher token usage is observed.”
We thank the reviewer for highlighting this trade-off. On Online-Mind2Web, PACT improves the success rate from 29.5% to 39.0% (+9.5 percentage points), while increasing token consumption by 15.4% and execution steps by 7.7%.
Although PACT does not reduce absolute execution costs on this benchmark, the relative improvement in success rate (+32.2%) substantially exceeds the increase in token consumption (+15.4%), indicating a more favorable success-to-token trade-off. These results demonstrate that PACT's efficiency benefits vary across benchmarks, while its improvements in task completion extend to live websites. We have clarified this trade-off in the revised paper.
Author response for Reviewer e8Hg - 1
Thank you for your careful review and for your positive assessment of the motivation, evaluation, and integration study. We appreciate your suggestions on related work, comparisons with existing methods, and evaluation beyond WebArena.
We thank the reviewer for highlighting these closely related approaches. We agree that reusable skill generation, verification, and context-dependent availability have been explored in prior work. We have expanded Section 2 to discuss SkillWeaver and WebXSkill and clarify our distinctions.
Unlike approaches that construct reusable agent skills, PACT focuses on automatically generating WebMCP-compatible capabilities grounded in existing website functionality. Specifically, PACT derives tools from page states, DOM structures, and application logic, registers them according to live page conditions rather than URL patterns alone, and bounds their execution within page-native operations. These tools preserve existing website workflows and human-facing interactions, while leaving navigation, reasoning, and tool composition to the calling agent.
Our contribution therefore lies in the integrated generation, verification, and agent integration of page-coupled tools, rather than claiming novelty for the individual mechanisms.
We thank the reviewer for this suggestion. WALT provides both its own tool designs and a dedicated agent architecture. To ensure a more controlled comparison, we integrated WALT tools into CUGA and evaluated both WALT and PACT tools using the same agent (CUGA), reasoning model (Qwen-3.7-Plus), and 812 WebArena tasks.
Comparison with WALT. Results are reported in Appendix A.15 (Table 23).
Under the same CUGA agent, PACT outperforms WALT tools by 3.9 percentage points while reducing mean token usage by 23.9%. We additionally report WALT with its original agent architecture for reference, although differences in agent design prevent a direct attribution of performance differences to the tools alone. These results are summarized in Section 4.1.2.
We additionally evaluated PACT on live websites using Online-Mind2Web (Appendix A.16). The results are summarized below.
Online-Mind2Web. AgentOccam with Qwen-3.7-Plus is evaluated on 210 tasks after excluding 90 task pairs affected by external errors.
PACT improves task success by 9.5 percentage points, while increasing mean token usage by 15.4% and steps by 7.7%. Unlike the locally hosted WebArena environments, live websites can introduce additional interaction overhead, and available tools may not cover every required operation. Tool selection, invocation, and subsequent browser interactions may therefore increase resource consumption. The higher success rate may also reflect additional progress on tasks that would otherwise terminate unsuccessfully, although we have not isolated these effects.
These findings demonstrate that PACT can improve task success on diverse live websites, while highlighting a trade-off between task completion and interaction cost. We do not claim exhaustive page-state coverage, and improving exploration efficiency and tool coverage for large dynamic websites remains an important direction for future work.
The revised Section 4.1.6 distinguishes WebArena Verified from the live-web evaluation, while Section 4 introduces the separate mock-site extension. WebArena Verified provides a check on revised task descriptions and evaluation within the WebArena task family. The mock-site extension examines generated websites, while the new Online-Mind2Web experiment evaluates public live websites.