acceptodds
Under review as a conference paper at ICLR 2027

Same Website, Different Implementations: Rethinking Robustness of Web Agents

Abstract

Web agents are increasingly capable of completing web tasks, but the diversity of real-world websites remains a major challenge to reliable deployment beyond controlled environments. One important but underexplored factor is website implementation: different implementations of the same functionality can provide similar user experiences while exposing agents to substantially different structural, textual, and visual signals. Existing evaluations do not account for such variation, making it unclear whether an agent can preserve its task-solving capability across functionally equivalent implementations. We introduce WebVariants, a controlled evaluation framework with nine websites, five implementations each, and 228 tasks shared across implementations, yielding 1,140 task–environment instances. Extensive evaluation reveals substantial sensitivity to implementation choice. Text- and screenshot-based agents exhibit different vulnerabilities, while richer multimodal observations do not consistently improve robustness across implementations. Implementation also affects whether agents recognize task completion after reaching a successful state. The resulting performance variation substantially alters agent rankings, with up to 50% of pairwise orderings reversing across implementations. We further investigate how to improve robustness across implementations. Training on a single implementation generalizes well to other implementations of websites seen during training, but its gains become less consistent on unseen websites. Under a matched training budget, training across multiple implementations improves robustness on unseen websites. Within this budget, covering more unique tasks is more effective than exposing fewer tasks to multiple implementations. Our results establish website implementation as an important dimension of both evaluation and training for robust web agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
web agents, robustness, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9Wz2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark to evaluate the robustness of web agents across functionally equivalent implementations of different websites. Results show that implementation differences significantly affect agent performance, rankings, and task-completion recognition, while multimodal observations do not consistently improve robustness. The paper also shows that training across diverse implementations improves generalization to unseen websites, and training on more different tasks is more effective than repeating the same tasks across multiple versions.

Critical strengths
  1. The paper studies how different website implementations affect the performance of web agents, which has not been widely explored in previous research.
  2. The paper introduces a benchmark with 9 websites, 5 versions of each website, and 228 tasks. It also tests different types of web agents.
  3. The paper shows that training on different website versions helps agents perform better on new websites. It also finds that training on more unique tasks is more effective.
Critical weaknesses
  1. The paper only considers four implementation variants, which may not fully reflect the differences between real-world websites. Most of these changes intentionally remove useful information from the webpage, so it is not surprising that agents perform worse. Also, the Icon-UI and Flat-UI variants change how the website looks, rather than just how it is implemented. The paper would be stronger if it tests different versions of the same website built independently by different developers.
  2. The paper checks that tasks can be completed across all website variants, but it does not show whether these versions are equally difficult for humans to use. For example, replacing text labels with icons or making buttons less noticeable could make a website harder to navigate. Without testing how well and how quickly humans complete tasks on each version, it is difficult to tell whether the differences in agent performance are caused by weaknesses in the agents or by some website versions being harder to use.
  3. The paper shows that agent rankings can change significantly across different website implementations. However, the analysis only includes nine relatively small agents, and several have very low success rates on default implementations. When the differences in success rates are small, agent rankings can change even if there is no meaningful difference in their performance. Also, some of the reported confidence intervals for Kendall's τ\tau are quite wide, making the results less certain. The paper should provide more statistical analysis to show whether these ranking changes are meaningful.
  4. The training experiments in Section 5 use only two unseen websites, which is too limited to support broad conclusions about how well the models perform on new websites. Both websites also come from the same WebArena-Infinity benchmark, so they may share similar designs and structures with the training websites. As a result, the improvements may be specific to these two websites rather than applying to other websites in general. Testing on more unseen websites and using different combinations of training and test websites would make the findings more convincing.
  5. The paper compares text-based and screenshot-based agents, but these agents differ in several ways beyond how they view websites. For example, screenshot-based agents must identify where to click on the screen, while text-based agents can use element IDs to interact with buttons and links directly. Therefore, some performance differences may come from how agents interact with websites rather than how well they handle different website implementations. A fairer comparison would test different ways of viewing and interacting with websites separately to better understand what causes the performance differences.
  6. The paper shows that agents often fail to stop even after successfully completing a task. The paper attributes the reason to that agents have trouble recognizing when a task is finished. However, an agent might recognize that the task is complete but continue because it is unsure or wants to double-check its work. The paper does not examine these different reasons. It would be helpful to test whether agents can correctly identify when a task is complete, separately from whether they decide to stop.
Questions for authors
  1. Do the four implementation variants adequately represent differences between real-world websites? Since some variants intentionally remove useful information or change the visual interface, how does the paper ensure that the observed performance drops reflect agent robustness rather than increased task difficulty?
  2. Given that Section 5 evaluates only two unseen websites from the same benchmark, how does the paper support the claim that multi-implementation training improves generalization to new websites?
  3. Since several evaluated agents have low success rates, how does the paper demonstrate that the reported ranking changes are statistically meaningful rather than caused by small performance differences or evaluation uncertainty?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 9Wz2,

Thank you again for your time and effort to provide your thoughtful and detailed review. They helped us strengthen the paper. As there are only 2 days left in the discussion phase, we would like to gently follow up on our responses. If anything seems unclear or you would like us to dive deeper into a specific part, we are happy to follow up. We look forward to hearing your thoughts!

Reviewer_3i8fOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors of this paper evaluate web agent sensitivity to website implementation by proposing a set of website variations for nine websites selected from WebArena-Infinity. When evaluating tasks across all website variations, they identify that models often underperform on such variations compared to the original implementations, and demonstrate that by finetuning on successful human trajectories, performance improves on unseen website variations. Agent generalization, however, is much weaker on held-out websites. They also demonstrate that modifying the observation interface does not improve robustness and task coverage, and that models exhibit high variance with respect to recognizing task completion.

Critical strengths
  1. I think the paper presents a new lens for studying web agent robustness, demonstrating that task “success” may be more brittle to implementation than has previously been thought. The authors demonstrate this lack of robustness through comprehensive experiments across multiple model families and types, which supports their case that implementation sensitivity is a persistent failure mode across multiple agents.
  2. I think that their study of web agent generalization shows some interesting findings, specifically that training on multiple implementations leads to better generalization on unseen websites. The authors’ evaluation of agent generalization capabilities is fairly comprehensive, and provides valuable insights for researchers studying web agent training.
Critical weaknesses
  1. I am concerned about the longevity of the benchmark given the straightforward approaches taken to construct the variants. While the GPT-5.6-Luna and Gemini-3.5-Flash-Lite results suggest that recent LLMs underperform on website variations in a similar manner to smaller web agent models on Div-25% and Poor-A11y, the models maintain their performance on Icon-UI and Flat-UI, suggesting that recent models are more likely to be robust to those variants.
  2. The evaluation scale is fairly limited, restricted to 228 evaluation tasks across nine websites. Consequently, the reported results may not generalize at scale.
Questions for authors
  1. How sensitive are the results in Table 1 to the agent framework used? The authors report using BrowserGym for interacting with the open-weight models, but does task difficulty change under other opensource frameworks (e.g. BrowserUse)?
  2. What is the agent framework used during LLM web agent evaluations (e.g. are all evaluations conducted on ReAct-style agents, or was a different framework used)?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 3i8f,

Thank you again for your time and effort to provide your thoughtful and detailed review. They helped us strengthen the paper. As there are only 2 days left in the discussion phase, we would like to gently follow up on our responses. We hope our responses have adequately addressed your questions and concerns. If any points remain unclear or require further clarification, we would be happy to provide additional details.

We sincerely appreciate your time and consideration.

Thank you for the updated frontier model analysis. The additional experiments with BrowserUse provide additional confidence that the results are platform agnostic, and I think they greatly strengthen the paper. My only remaining concern relates to benchmark size and diversity: having only 9 websites and 456 tasks is still limiting compared to open-web evaluation systems, that expose agents to a wider variety of dynamic websites and tasks.

As it stands, I will maintain my positive recommendation and score, and recommend the paper for acceptance.

Reviewer_5W3LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WebVariants - benchmark that tests how web agents perform when the same websites are presented through different but functionally equivalent interfaces. It evaluates agents on 228 tasks across nine websites and five interface variants, finding large performance differences depending on the implementation, including when the underlying task remains unchanged. Agent rankings also change to high extent across variants. The authors show that training on multiple interface variants improves generalisation to unseen websites, although adding more unique tasks is more beneficial than adding more variants of the same tasks.

Critical strengths

The benchmark isolates the effect of interface design. I think the substantial strength is in isolation of the effect, as each task uses the same backend verifier and is verified across five implementations, thanks to which the performance differences can be attributed to the changes in the interface interface and not e.g. the task difficulty. The structural and accessibility variants also seem to cover gaps in existing benchmarks, providing the novelty.

Transparency of the evaluation Apart from the overall success rates, authors examine consistency across trials, robustness, changes in agent rankings, recognition of task completion, and the effects of different training strategies, alongside with the comprehensive appendix. This allows for recognition of limitation for future work and follows best practices in evaluations.

Critical weaknesses

Reporting errors and insufficient statistical evidence weaken the main claims. Section 5 appears to use the wrong denominator for source-website results. For example, Qwen3.5-4B achieves 40/228 successes overall, split into 32 source and 8 held-out successes. However, Table 12 divides the 32 source successes by 228 rather than 176 tasks. This error affects Table 12, Figures 5–7, and the reported training gains, making source-versus-held-out comparisons unreliable. I believe this is the major weakness, which impacts my score. The abstract also highlights Kendall’s τ = 0.00, but this is the lowest correlation among ten comparisons. No confidence intervals are reported for Pass^3 or F1, the paired benchmark design is not used for statistical testing, and all fine-tuning experiments use a single run. I'm worried that with only 52 held-out tasks, several reported improvements may reflect random variation rather than meaningful gains.

Issues with interface consistency and training comparisons Icon-UI and Flat-UI should theroetically leave the accessibility tree largely unchanged, but they produce substantial performance differences even for text-only agents. This suggests that the implementations differ in ways not captured by the proposed taxonomy. My worry is also that some structural changes also remove interactive elements from the accessibility tree, making tasks inaccessible rather than simply harder to interpret.

It seems that the training comparison has a similar issue and multi-implementation models are evaluated on interface variants they have already seen during training. Particularly for Gemma, when both the website and implementation are held out, the benefits become less consistent. The multi-implementation model trained without default also performs worse than the default-only baseline (even though the difference is not that large). This weakens the claim that implementation diversity consistently improves generalisation.

Questions for authors

As mentioned before, I would prioritise the first weakness.

  1. In Table 12 and the source-website panels of Figures 5 to 7, are Pass^3 and C computed over the 176 source-website tasks or over all 228?
  2. For each excluded implementation E^h, what are Pass^3 and F1 on the held-out websites when evaluated only on E^h, for the Multi-Implementation model versus (a) the mean of the four Single models and (b) the Default-only model, with task-level bootstrap CIs?
  3. How much does the accessibility tree differ between Default, Icon-UI and Flat-UI (node and token counts, fraction of nodes changed), and how much does the rendered page differ between Default, Div-25% and Poor-A11y? What explains the consistent T-only movement under Icon-UI and Flat-UI, given that these variants are meant to leave the tree unchanged? Relatedly, what fraction of task-critical targets lose a BID-bearing interactive node under Div-25% and Poor-A11y, and can agents still reach those targets through StaticText or generic nodes?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 5W3L,

Thank you again for your time and effort to provide your thoughtful and detailed review. They helped us strengthen the paper. As there are only 2 days left in the discussion phase, we would like to gently follow up on our responses. We hope you had a chance to review our responses, and we would be happy to address any remaining questions or concerns before the discussion period ends.

We appreciate your time and consideration and look forward to hearing your thoughts.

Reviewer_5W3LOpenReview ↗

Happy to raise the score if the remaining points will be addressed

Thank you for your comments and preparing the results and analysis I asked for. The new CIs, additional splits, default-only comparison and BID verification address most of my concerns, and I'd be happy to raise my score. Just small two points:

  • I believe the denominator correction appears to be incomplete. In the revised Table 15, trained-model rows use 176 tasks, however the base-model rows still use 228 (Qwen3.5-4B base T+V on Default: 14.04 = 32/228, should be 18.18). It seems that figure 5 subtracts these, so it now overstates gains (D_Default/E_Default: +18.35 rather than +14.21). The 8.25 quoted for Qwen3.5-4B matches the original all-228 computation exactly, with consistent denominators I get about 10.7 (Qwen) and 6.5 (Gemma). In this setting the conclusion still holds, and I have certainty at around 80% that this is incomplete, but please revise it and correct Table 15, Figure 5 and Section 5.2.
  • Please confirm that skip_generic=False was used in every reported experiment, and not only in the post-hoc verification. It would be useful to document it clearly in the Appendix.

And for the final version, it would be very helpful if you could please:

  • instead of leading with 0.00, present the range of τ,
  • adjust the Section 4.2 attribution to account for that Flat-UI changes the AXTree more than Div-25%
  • state that fine-tuning results are single runs,
  • and check Table 2, which currently gives same CIs for Icon-UI and Flat-UI.
AuthorsOpenReview ↗

Official Comment by Authors

Thank you very much for your encouraging response and for carefully reviewing our revisions. We sincerely appreciate for catching the incomplete denominator correction. You are correct that the base-model results were still computed using 228 tasks, while the trained-model results used 176 tasks. We have now corrected this inconsistency and updated Table 16, Figures 5 and 11, and Section 5.2 accordingly. The corrected average gains on source websites are 10.68% for Qwen3.5-4B and 6.53% for Gemma-4-E2B-it, further supporting our original conclusion. Regarding skip_generic=false, we have explicitly documented this in Appendix D.3.

We have also incorporated your additional suggestions. We revised the Figure 1 caption to clarify that Kendall's τ=0.00\tau=0.00 represents one example of implementation-dependent ranking variation and to report the full range of τ\tau across implementation pairs. In Section 4.2, we added a footnote clarifying that Icon-UI and Flat-UI also introduce changes to accessibility tree observations, with a quantitative comparison provided in Appendix E.8 (Table 14). We additionally clarified in Appendix D.4.2 that each fine-tuning configuration was trained once. Finally, we rechecked the confidence intervals in Table 2 that the identical intervals reported for Icon-UI and Flat-UI are correct.

Thank you again for your thoughtful and detailed feedback throughout the review process. Your comments have been very helpful in improving the accuracy of our paper, and we greatly appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.