acceptodds
Withdrawn from ICLR 2027

WebMCPWorld: Can LLMs Synthesize WebMCP Tools to Finish Open-Ended Tasks in Web APPs?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) have demonstrated remarkable capability at both writing code and operating computers. In real-world scenarios, users make colloquial requests and let an agent finish computer-use tasks; for a frequently repeated workflow, they prefer to build a reusable tool(e.g., a skill, an MCP server), so the work runs stably instead of being re-derived. Existing works prove that agents given MCP or CLI tools alongside GUI actions succeed more often and in fewer steps. Yet benchmarks have not kept pace: SWE-bench scores a patch to a named repository, application-generation benchmarks assess a model's ability to generate complete applications, and computer-use benchmarks measure GUI agents' success rates on individual computer-use tasks. None of them evaluates this ability—turning a colloquial request into a small, working, reusable tool over an application that already exists. To address this gap, we introduce WebMCPWorld, a benchmark consisting of 50 real-world web applications and 1280 colloquial user queries. Given the source code of a web application and a query, a model must do two things: locate the application functionality that the query targets, and turn that part of its source into the WebMCP tool layer, and declare a parameter interface that a runtime must later bind the query's implicit arguments to, without ever observing that the runtime or its inputs. An execution-based runtime judges the synthesized tooling both when supplied with the query's true parameters and on whether the user's query is ultimately fulfilled. WebMCPWorld remains highly challenging for current frontier models. Across 18 configurations of 10 LLMs, even the best-performing setup, Claude Opus 5 with Claude Code, achieves a mean end-to-end completion rate of only 10.0% across all queries. These results highlight the difficulty of turning colloquial requests into reliable, reusable tools for existing web applications.

Primary area:
datasets and benchmarks
Keywords:
Tool-synthesis, Benchmark, LLM agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Pc4eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WebMCPWorld, a benchmark that tests how well language models are at creating their own browser based WebMCP tools for a given application. The idea is to address a gap in current evaluation of agent capability - often when benchmarking web use by agents, the agents are provided with the tools needed to solve the problem. This benchmark instead measures how the agent would do if it was just given the user query and the ability to create whatever tool it needs to complete the task. To do this, they curate a benchmark of over 50 popular open source web applications, and ~1.28k user queries, and set the agents to the task of addressing the queries by first creating a webMCP tool from the source of the web application.

The 50 applications are curated from open source repos on GitHub, filtering for popular (at least 50 stars), sufficiently complex w/ the actual business features covered, access to the corresponding source code, support for local deployment. The candidate repos are screened by an LLM to ensure that they are good applications to create sufficiently complex user queries for, and then a GUI agent is used to explore the business features exposed by the application to record the interactions and convert these actions sequences into tasks and tests. Human reviewers are used to confirm that the features and corresponding tasks are valid, and also review the tests.

Next, the tasks are run through the actual evaluation pipeline, where the application is deployed with a seeded state - and a synthesis agent makes use of the source code and task queries to create the workflows to be executed using the WebMCP tools (including defining the interface, contract, code etc.). They then use these tools in a sandbox with the right parameters at run-time. They first run an execution test to ensure that the workflow is valid (does the right thing), and then run an end to end test, where the agent tries to use these workflows with the provided query to solve the task.

Judging is done for both tests: the final state recorded is checked agianst the expected final state for the workflow test. For the e2e test, actual task completion is checked using an automated judge that has access to the query, the state, the traces and the success criteria.

10 different models are evaluated. Runs are repeated 3 times. Two different metrics are tracked - query to successful completion rate (% of all queries that succeeded), and agent success rate (% of retained agent state trials that succeeded). The difference in the two rates is that the ASR excludes the queries that don't pass the workflow execution stage to the agent stage.

The results show that there is still significant headroom for models to improve on this task, as even the highest mean QCSR is only at 10%. The results also show a significant difference between the highest and the second (10% to 4.3%), which is pretty interesting.

The authors also do further analysis on task difficulty proxied by the number of GUI steps required (no pattern that maps to success rate), reuse (most variants do not succeed)/repeatability (repeated success was rare, failure on repeatition of tool use was more common), and failure patterns.

Critical strengths
  • The problem statement proposed and the task that is being measured is a significant and useful task, especially as agents get more capable and developers are relying more and more on AI to help build out their systems.
  • Particularly happy with how the tasks themselves are curated, along with the human review that goes into making sure that the features are actually accessible via the GUI and that the tests are reviewed before the execution phase.
Critical weaknesses
  • Motivation for the work is not very crisp - the main issue is that the abstract introduces this from the end-user being someone trying to use the web-service, however the need for the source code makes this better suited for the developers of the service who are trying to make it more agent friendly.

  • The judges are not calibrated with human judgement / no human alignment numbers are provided. While human review of the test success criteria is mentioned during task creation, there isn't any analysis on how often things needed correction (less important), or how often the agents failed at checking these criteria (much more important). If the judge model was just poor at figuring out whether the output state is correct or not, the correctness of the criteria themselves is not going to help. I think this is the primary pain point with the paper - it's hard to take the results at face value when there's limited evidence on how robust the judging system is.

  • Harness budget differences were called out as a limitation, however this actually makes the ranking claims a little weaker; this might not be something that's easily fixed though, just as a function of how different the rankings are. Would be interesting to see how all the models perform on an open source harness, if possible.

Questions for authors
  1. See question about motivation in weaknesses - could that be rephrased better to make it more clear who the target audience is? [clarification]

  2. Do you have any analysis on the effectiveness/validity of the judge model? Doing a blind human review on the outcome and seeing if this lines up with how the judge decided if the task was completed would go a long way in validating the results here.

  3. On the repeatability/reuse analysis work - it seems like your experiments are regenerating the tool every time, in addition to execution+eval - is there a way to freeze the generated tools and test them on unseen queries? This will give us a control for whether the drop in the success rate is because of tool generation vs variance in tool use.

Reviewer_7uNgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a model can turn a colloquial request plus an app's source code into a reusable tool layer — definitions, workflow code, a self-designed parameter contract — without reference schemas. WebMCPWorld supplies 50 self-hostable applications, 258 features and 1,280 queries, scored by a pinned workflow test with ground-truth arguments and by an end-to-end agent restricted to the synthesized tools. Ten models in 18 configurations are run; the best, Claude Opus 5 with Claude Code, reaches 10.0% mean completion against 4.45% for GUI-only, and 95.0% of groups hold no successful end-to-end query.

Critical strengths

The task is new and the construction protocol explicit (Section 3.1, Appendix A). Design discipline is good: reference schemas are withheld (Appendix C.1), only the first query per feature drives synthesis, and an optimistic success flag counts as failure. The bookkeeping is careful: Table 9 reconciles all 69,120 planned cells, QCSR and ASR use distinct denominators, and Section 5 concedes confounding rather than hiding it.

Critical weaknesses

The comparison moves several things at once. GUI-only runs DeepSeek V4.1 Flash at 60 steps per query, while the 10.0% figure is Claude Opus 5 at five turns. Within one model the harness alone moves QCSR by 3-5x (Qwen3.8 Max: 0.9 / 2.7 / 4.3 for CC / CX / MSA), which is larger than the effect being reported, and it is not explained by budget enforcement alone, since Codex sets no code-level limit yet does not lead. The only model-matched pair, MSA-DeepSeek, is 4.3% against 4.45% for GUI-only. Section 3.3 also reserves clicking for the GUI baseline, so combining tools with GUI operation is not evaluated.

The synthesized artifact manipulates the DOM rather than calling an application interface. The workflow body is "DOM interaction logic" in page context (Appendix C.1.1), with harness helpers such as clash_upload_file because page JavaScript cannot set a file input, and applications whose backend routes appear in the source are not driven through them. The evaluated contrast is therefore between a script written in advance and one decided live. Appendix C.3.1 also notes that conformance to WebMCP is not claimed.

There is no ceiling and no positive control: no author-written reference workflow, and no results for the schema-supplied condition of Appendix C.1.3. The pinned test reaches 8.4% even with ground-truth arguments, and it scores only cells whose reference parameters map one-to-one, so 45.2% of planned cells receive no workflow verdict.

The main metric has little external anchoring. Only 3,658 of 34,924 agent cells also lie in the workflow-eligible set (Appendix D.4), and all labels come from one LLM judge that is also the runtime agent; Section 5 notes the resulting bias is untested.

Provenance and reuse are not reported. Human review is mentioned without annotator counts or agreement statistics, and tools are re-synthesized at every repetition (Appendix D.5), so no fixed artifact is re-executed.

Questions for authors

See weakness

Reviewer_s2vUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces WebMCPWorld, a benchmark for turning a colloquial request and the source code of a web application into a reusable WebMCP tool layer. It contains 50 applications, 258 features and 1280 queries. A model must locate the target feature, synthesize the tool and declare a parameter interface. Evaluation has two modes, a workflow test with reference arguments and an end-to-end agent test. Ten LLMs in 18 configurations are evaluated. The best mean query-to-completion rate is 10.0%.

Critical strengths

The task is new and practically relevant.

Critical weaknesses

The withheld binding contract makes the target partly ill-posed.

The evaluation depends on an uncalibrated LLM judge and a metric that mixes failure stages.

Contamination and query naturalness are not measured.

No simple baseline is included, such as retrieval of existing endpoints or a template wrapper. The comparison is only across model and harness configurations.

I can't understand the picture 2 and it's meaningless.

Questions for authors

Can the synthesis model see the binding tool schema? If not, how can it declare a compatible interface?

How were the colloquial queries generated and validated? Do they leak feature names?

What is the human agreement rate with the automated judge on a sample?

How does a simple retrieval or template baseline perform on the same tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.