- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces WebMCPWorld, a benchmark that tests how well language models are at creating their own browser based WebMCP tools for a given application. The idea is to address a gap in current evaluation of agent capability - often when benchmarking web use by agents, the agents are provided with the tools needed to solve the problem. This benchmark instead measures how the agent would do if it was just given the user query and the ability to create whatever tool it needs to complete the task. To do this, they curate a benchmark of over 50 popular open source web applications, and ~1.28k user queries, and set the agents to the task of addressing the queries by first creating a webMCP tool from the source of the web application.
The 50 applications are curated from open source repos on GitHub, filtering for popular (at least 50 stars), sufficiently complex w/ the actual business features covered, access to the corresponding source code, support for local deployment. The candidate repos are screened by an LLM to ensure that they are good applications to create sufficiently complex user queries for, and then a GUI agent is used to explore the business features exposed by the application to record the interactions and convert these actions sequences into tasks and tests. Human reviewers are used to confirm that the features and corresponding tasks are valid, and also review the tests.
Next, the tasks are run through the actual evaluation pipeline, where the application is deployed with a seeded state - and a synthesis agent makes use of the source code and task queries to create the workflows to be executed using the WebMCP tools (including defining the interface, contract, code etc.). They then use these tools in a sandbox with the right parameters at run-time. They first run an execution test to ensure that the workflow is valid (does the right thing), and then run an end to end test, where the agent tries to use these workflows with the provided query to solve the task.
Judging is done for both tests: the final state recorded is checked agianst the expected final state for the workflow test. For the e2e test, actual task completion is checked using an automated judge that has access to the query, the state, the traces and the success criteria.
10 different models are evaluated. Runs are repeated 3 times. Two different metrics are tracked - query to successful completion rate (% of all queries that succeeded), and agent success rate (% of retained agent state trials that succeeded). The difference in the two rates is that the ASR excludes the queries that don't pass the workflow execution stage to the agent stage.
The results show that there is still significant headroom for models to improve on this task, as even the highest mean QCSR is only at 10%. The results also show a significant difference between the highest and the second (10% to 4.3%), which is pretty interesting.
The authors also do further analysis on task difficulty proxied by the number of GUI steps required (no pattern that maps to success rate), reuse (most variants do not succeed)/repeatability (repeated success was rare, failure on repeatition of tool use was more common), and failure patterns.
Critical strengths
- The problem statement proposed and the task that is being measured is a significant and useful task, especially as agents get more capable and developers are relying more and more on AI to help build out their systems.
- Particularly happy with how the tasks themselves are curated, along with the human review that goes into making sure that the features are actually accessible via the GUI and that the tests are reviewed before the execution phase.
Critical weaknesses
-
Motivation for the work is not very crisp - the main issue is that the abstract introduces this from the end-user being someone trying to use the web-service, however the need for the source code makes this better suited for the developers of the service who are trying to make it more agent friendly.
-
The judges are not calibrated with human judgement / no human alignment numbers are provided. While human review of the test success criteria is mentioned during task creation, there isn't any analysis on how often things needed correction (less important), or how often the agents failed at checking these criteria (much more important). If the judge model was just poor at figuring out whether the output state is correct or not, the correctness of the criteria themselves is not going to help. I think this is the primary pain point with the paper - it's hard to take the results at face value when there's limited evidence on how robust the judging system is.
-
Harness budget differences were called out as a limitation, however this actually makes the ranking claims a little weaker; this might not be something that's easily fixed though, just as a function of how different the rankings are. Would be interesting to see how all the models perform on an open source harness, if possible.
Questions for authors
-
See question about motivation in weaknesses - could that be rephrased better to make it more clear who the target audience is? [clarification]
-
Do you have any analysis on the effectiveness/validity of the judge model? Doing a blind human review on the outcome and seeing if this lines up with how the judge decided if the task was completed would go a long way in validating the results here.
-
On the repeatability/reuse analysis work - it seems like your experiments are regenerating the tool every time, in addition to execution+eval - is there a way to freeze the generated tools and test them on unseen queries? This will give us a control for whether the drop in the success rate is because of tool generation vs variance in tool use.