- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark to evaluate the robustness of web agents across functionally equivalent implementations of different websites. Results show that implementation differences significantly affect agent performance, rankings, and task-completion recognition, while multimodal observations do not consistently improve robustness. The paper also shows that training across diverse implementations improves generalization to unseen websites, and training on more different tasks is more effective than repeating the same tasks across multiple versions.
Critical strengths
- The paper studies how different website implementations affect the performance of web agents, which has not been widely explored in previous research.
- The paper introduces a benchmark with 9 websites, 5 versions of each website, and 228 tasks. It also tests different types of web agents.
- The paper shows that training on different website versions helps agents perform better on new websites. It also finds that training on more unique tasks is more effective.
Critical weaknesses
- The paper only considers four implementation variants, which may not fully reflect the differences between real-world websites. Most of these changes intentionally remove useful information from the webpage, so it is not surprising that agents perform worse. Also, the Icon-UI and Flat-UI variants change how the website looks, rather than just how it is implemented. The paper would be stronger if it tests different versions of the same website built independently by different developers.
- The paper checks that tasks can be completed across all website variants, but it does not show whether these versions are equally difficult for humans to use. For example, replacing text labels with icons or making buttons less noticeable could make a website harder to navigate. Without testing how well and how quickly humans complete tasks on each version, it is difficult to tell whether the differences in agent performance are caused by weaknesses in the agents or by some website versions being harder to use.
- The paper shows that agent rankings can change significantly across different website implementations. However, the analysis only includes nine relatively small agents, and several have very low success rates on default implementations. When the differences in success rates are small, agent rankings can change even if there is no meaningful difference in their performance. Also, some of the reported confidence intervals for Kendall's are quite wide, making the results less certain. The paper should provide more statistical analysis to show whether these ranking changes are meaningful.
- The training experiments in Section 5 use only two unseen websites, which is too limited to support broad conclusions about how well the models perform on new websites. Both websites also come from the same WebArena-Infinity benchmark, so they may share similar designs and structures with the training websites. As a result, the improvements may be specific to these two websites rather than applying to other websites in general. Testing on more unseen websites and using different combinations of training and test websites would make the findings more convincing.
- The paper compares text-based and screenshot-based agents, but these agents differ in several ways beyond how they view websites. For example, screenshot-based agents must identify where to click on the screen, while text-based agents can use element IDs to interact with buttons and links directly. Therefore, some performance differences may come from how agents interact with websites rather than how well they handle different website implementations. A fairer comparison would test different ways of viewing and interacting with websites separately to better understand what causes the performance differences.
- The paper shows that agents often fail to stop even after successfully completing a task. The paper attributes the reason to that agents have trouble recognizing when a task is finished. However, an agent might recognize that the task is complete but continue because it is unsure or wants to double-check its work. The paper does not examine these different reasons. It would be helpful to test whether agents can correctly identify when a task is complete, separately from whether they decide to stop.
Questions for authors
- Do the four implementation variants adequately represent differences between real-world websites? Since some variants intentionally remove useful information or change the visual interface, how does the paper ensure that the observed performance drops reflect agent robustness rather than increased task difficulty?
- Given that Section 5 evaluates only two unseen websites from the same benchmark, how does the paper support the claim that multi-implementation training improves generalization to new websites?
- Since several evaluated agents have low success rates, how does the paper demonstrate that the reported ranking changes are statistically meaningful rather than caused by small performance differences or evaluation uncertainty?
Official Comment by Authors
Dear Reviewer 9Wz2,
Thank you again for your time and effort to provide your thoughtful and detailed review. They helped us strengthen the paper. As there are only 2 days left in the discussion phase, we would like to gently follow up on our responses. If anything seems unclear or you would like us to dive deeper into a specific part, we are happy to follow up. We look forward to hearing your thoughts!