WebMCPWorld: Can LLMs Synthesize WebMCP Tools to Finish Open-Ended Tasks in Web APPs?

Large language models (LLMs) have demonstrated remarkable capability at both writing code and operating computers. In real-world scenarios, users make colloquial requests and let an agent finish computer-use tasks; for a frequently repeated workflow, they prefer to build a reusable tool(e.g., a skill, an MCP server), so the work runs stably instead of being re-derived. Existing works prove that agents given MCP or CLI tools alongside GUI actions succeed more often and in fewer steps. Yet benchmarks have not kept pace: SWE-bench scores a patch to a named repository, application-generation benchmarks assess a model's ability to generate complete applications, and computer-use benchmarks measure GUI agents' success rates on individual computer-use tasks. None of them evaluates this ability—turning a colloquial request into a small, working, reusable tool over an application that already exists. To address this gap, we introduce WebMCPWorld, a benchmark consisting of 50 real-world web applications and 1280 colloquial user queries. Given the source code of a web application and a query, a model must do two things: locate the application functionality that the query targets, and turn that part of its source into the WebMCP tool layer, and declare a parameter interface that a runtime must later bind the query's implicit arguments to, without ever observing that the runtime or its inputs. An execution-based runtime judges the synthesized tooling both when supplied with the query's true parameters and on whether the user's query is ultimately fulfilled. WebMCPWorld remains highly challenging for current frontier models. Across 18 configurations of 10 LLMs, even the best-performing setup, Claude Opus 5 with Claude Code, achieves a mean end-to-end completion rate of only 10.0% across all queries. These results highlight the difficulty of turning colloquial requests into reliable, reusable tools for existing web applications.