EgoXR-GUI: Benchmarking GUI Grounding in Physical–Digital Extended Reality

The development of multimodal large language models has significantly advanced intelligent graphical user interface (GUI) agents. However, current research and benchmarking mainly focus on traditional platforms, like computer desktop and mobile interfaces, without considering the rapidly developing field of extended reality (XR). Unlike traditional platforms, XR requires models to understand the vital link between digital overlays and complex physical contexts. Moreover, unique platform artifacts and real-world egocentric interactions introduce additional challenges, such as perspective distortion, hand occlusion, and cluttered backgrounds. To this end, we present EgoXR-GUI, the first XR-specific GUI grounding benchmark, spanning direct, spatial, and semantic grounding tasks, and featuring over 1,000 high-quality annotated samples collected across diverse real-world scenarios. Our results show that state-of-the-art models perform significantly worse on these tasks than on conventional 2D grounding, revealing a critical gap in their ability to reason within hybrid physical-digital environments.