Same Website, Different Implementations: Rethinking Robustness of Web Agents

Web agents are increasingly capable of completing web tasks, but the diversity of real-world websites remains a major challenge to reliable deployment beyond controlled environments. One important but underexplored factor is website implementation: different implementations of the same functionality can provide similar user experiences while exposing agents to substantially different structural, textual, and visual signals. Existing evaluations do not account for such variation, making it unclear whether an agent can preserve its task-solving capability across functionally equivalent implementations. We introduce WebVariants, a controlled evaluation framework with nine websites, five implementations each, and 228 tasks shared across implementations, yielding 1,140 task–environment instances. Extensive evaluation reveals substantial sensitivity to implementation choice. Text- and screenshot-based agents exhibit different vulnerabilities, while richer multimodal observations do not consistently improve robustness across implementations. Implementation also affects whether agents recognize task completion after reaching a successful state. The resulting performance variation substantially alters agent rankings, with up to 50% of pairwise orderings reversing across implementations. We further investigate how to improve robustness across implementations. Training on a single implementation generalizes well to other implementations of websites seen during training, but its gains become less consistent on unseen websites. Under a matched training budget, training across multiple implementations improves robustness on unseen websites. Within this budget, covering more unique tasks is more effective than exposing fewer tasks to multiple implementations. Our results establish website implementation as an important dimension of both evaluation and training for robust web agents.