- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents EgoXR-GUI, a benchmark for GUI grounding in physical–digital extended reality (XR) environments. It contains 1,068 annotated samples across 47 applications and interfaces, covering three tasks: Explicit, Spatial, and Semantic Grounding. The authors evaluate 28 existing models and explore synthetic-data fine-tuning to improve grounding performance.
Critical strengths
- The benchmark contributes valuable long-tail GUI grounding data by introducing real-world XR scenarios.
- The evaluation covers 28 models with different architectures and training paradigms, providing a useful baseline for future XR GUI grounding research.
Critical weaknesses
- Limited analysis of physical backgrounds. The paper repeatedly highlights complex physical backgrounds as a key challenge in XR grounding, but does not systematically examine their impact. Controlled experiments with varying background complexity would help clarify how much they contribute to grounding difficulty.
- Incomplete experimental results. The Abstract claims that existing models perform worse on EgoXR-GUI than on conventional 2D grounding tasks, but direct experimental results are not provided. Moreover, Table 4 only reports Semantic Grounding results for the fine-tuned model, leaving its performance on Explicit and Spatial Grounding, as well as the full benchmark, unclear.
- Limited comparison with existing XR benchmarks. Table 1 (Lines 108–117) only includes conventional Desktop, Web, and Mobile GUI benchmarks, omitting related XR works such as Orienter and GUI-World. Although discussed in Section 2.2.2 and Figure 2(a), these works deserve a more systematic comparison on table.
Questions for authors
- How does physical background complexity affect grounding performance? Could the authors provide controlled experiments to examine this?
- Can the authors provide a direct comparison with conventional 2D grounding and report the fine-tuned model's performance on all three task categories?
- Could the authors include existing XR benchmarks in Table 1 to better clarify the differences from prior work?
4 comments
Thank you for the additional experiments. The responses to W2 and W3 have addressed my concerns.
However, W1 remains partially addressed. The experiment only examines Explicit Grounding, leaving the impact of physical background complexity on Spatial and Semantic Grounding unexplored.
A complexity-stratified analysis across all three grounding task types, if possible, would help clarify the impact of physical backgrounds and better support the paper's motivation.
Thank you for confirming that W2 and W3 are addressed. We agree that the effect of physical backgrounds on Spatial and Semantic grounding is central to our motivation, and appreciate the suggestion to examine all task types. In an exploratory complexity-stratified analysis, all three 8B baselines score lower on Spatial and Semantic grounding in higher-clutter scenes. Removing the background also lowers Spatial and Semantic accuracy for all four models tested, with the largest Semantic drop for Gemini.
Spatial and Semantic instructions rely on physical context: spatial relations link physical referents to UI targets, and physical objects or their properties determine the intended option. Clutter can make this inference harder, while blurring or masking removes the context itself. We examine the two effects separately. Qwen, MAI and Venus denote Qwen3-VL-8B, MAI-UI-8B and UI-Venus-1.5-8B; all values are percentages and Δ is in percentage points.
1. Complexity-stratified analysis
We used GPT-6-Astra to rate task-irrelevant physical clutter on a 1–5 scale without access to model predictions or correctness, and averaged scores within each screenshot and task type. Using reviewed task labels, we compare low (≤2) with higher (>2) clutter; accuracy is instruction-weighted.
| Task (n low / higher) | Model | Low | Higher | Δ |
|---|---|---|---|---|
| Spatial (29 / 121) | Qwen | 55.17 | 40.50 | −14.68 |
| Spatial | MAI | 58.62 | 53.72 | −4.90 |
| Spatial | Venus | 62.07 | 52.89 | −9.18 |
| Semantic (8 / 113) | Qwen | 50.00 | 17.70 | −32.30 |
| Semantic | MAI | 37.50 | 28.32 | −9.18 |
| Semantic | Venus | 50.00 | 28.32 | −21.68 |
Higher clutter is associated with 4.90–14.68 pp lower Spatial and 9.18–32.30 pp lower Semantic accuracy for all three models.
Place–activity pairs illustrate this. For Spatial, living room × exercise centers on curtains, radiators and a treadmill, whereas living room × daily routine shows wider room views with furniture, decorations and dispersed windows (median background edge density 0.230% vs. 0.949%). For Semantic, restaurant × ordering food usually shows an isolated food item, whereas supermarket × shopping shows dense shelves of similar products (2.166% vs. 7.115%).
| Model | Exercise (n=20) | Daily routine (n=29) | Δ | Restaurant (n=9) | Supermarket (n=15) | Δ |
|---|---|---|---|---|---|---|
| Qwen | 70.00 | 31.03 | −38.97 | 33.33 | 13.33 | −20.00 |
| MAI | 85.00 | 55.17 | −29.83 | 55.56 | 6.67 | −48.89 |
| Venus | 75.00 | 72.41 | −2.59 | 33.33 | 20.00 | −13.33 |
All three models also score lower in the more complex pair: by 2.59–38.97 pp on Spatial and 13.33–48.89 pp on Semantic.
These are exploratory associations, not causal evidence. The low-clutter Semantic group has only eight instructions, tests on the continuous score were not significant, other scoring dimensions gave mixed trends, and the paired subsets also differ in task and UI content (and, for Semantic, in place).
2. Background removal
We compare original images, Gaussian blur (σ = 30 px) and uniform gray (RGB 128), preserving the UI; instructions, target boxes, labels and inference settings are fixed.
| Model | Task | Original (= Table 2) | σ30 | Δσ30 | Gray | ΔGray |
|---|---|---|---|---|---|---|
| Qwen | Spatial | 51.14 | 38.42 | −12.72 | 36.16 | −14.98 |
| Qwen | Semantic | 23.17 | 22.56 | −0.61 | 22.56 | −0.61 |
| MAI | Spatial | 64.77 | 45.76 | −19.01 | 45.76 | −19.01 |
| MAI | Semantic | 30.49 | 18.90 | −11.59 | 21.34 | −9.15 |
| Venus | Spatial | 69.89 | 49.15 | −20.74 | 46.89 | −23.00 |
| Venus | Semantic | 28.66 | 20.12 | −8.54 | 23.17 | −5.49 |
Because these baselines start from low Semantic accuracy, we added Gemini 3.1 Flash, a newer release than the Gemini 3 Flash Preview in Table 2. Each intervention is compared with Gemini 3.1 Flash’s own original-image run; the Table 2 results are unchanged.
| Gemini 3.1 Flash | Original | σ30 | Δσ30 | Gray | ΔGray |
|---|---|---|---|---|---|
| Explicit | 86.38 | 89.13 | +2.75 | 89.41 | +3.03 |
| Spatial | 67.23 | 59.89 | −7.34 | 53.11 | −14.12 |
| Semantic | 65.85 | 41.46 | −24.39 | 34.76 | −31.10 |
Gray lowers Spatial and Semantic accuracy for all four models, by 5.49–31.10 pp on Semantic. Gemini shows the largest Semantic drop (31.10 pp), with a 14.12 pp Spatial drop and a small Explicit gain, underscoring the role of physical context in background-dependent reasoning.
Spot-checks explain the remaining accuracy: instruction or UI text, small candidate sets and selection biases offer alternative cues. In Semantic #1003, the selected “iced” option remains after the drink label is removed; in Spatial #695, the timer windows keep their left–right arrangement after the pots disappear. Segmentation also leaves some physical fragments around protected or translucent UI, so gray does not remove every usable cue.
In the revision, we will add both analyses. For the camera-ready version, we will also collect matched simple and complex backgrounds for the same tasks, preserving the relevant physical cues and UI, to isolate the effect of complexity.
Thank you again for this constructive suggestion, which has helped us clarify how physical backgrounds affect each grounding task.
Thank you for the thoughtful and detailed response, which addresses most of my concerns.
However, there is still room for improvement. While EgoXR-GUI introduces a valuable benchmark for an important XR GUI grounding problem, it does not yet fully explain what makes this setting particularly challenging. The dataset is also relatively limited in scale. Additionally, polygon-based annotations may be more appropriate for XR interfaces, and the visual quality and clarity of the figures could be further improved.
Therefore, I decide to maintain my current rating, although my overall assessment remains borderline between Weak Reject and Weak Accept.
Reply to Reviewer KkEo
We thank the reviewer for recognizing our real-world XR data and 28-model baseline, and address each concern in order.
W1 / Q1. Background intervention
Our controlled experiment shows that, for these models and conditions, weakening the scene around the UI changes Explicit accuracy only slightly, so complex backgrounds are not a major interference for high-accuracy models.
Explicit instructions name the target directly, isolating scene interference. On all 727 (206 screenshots), we fixed screenshot, instruction, box and inference settings and altered only pixels outside a mask preserving the UI, target boxes and translucent-panel content: Gaussian blur (σ = 4, 8, 16, 30 px) or uniform gray (RGB 128). Clicks inside the box count as correct.
Unedited predictions matched Table 2 hit for hit for Qwen3-VL-8B and UI-Venus-1.5-8B; MAI-UI-8B differed by two hits (643 vs. 641/727), so changes use the unedited run. Changes ranged from −0.28 to +1.65 pp, and all 15 simultaneous 95% intervals (screenshot-level paired bootstrap, 50,000 draws) included zero; the largest was UI-Venus-1.5-8B with gray, +1.65 pp [−0.54, +3.85].
By design, Spatial and Semantic instructions refer to physical objects or positions, so the scene matters more there.
We will add this experiment with more baselines to the paper and narrow the claim that visual clutter impairs Explicit grounding.
W2(i) / Q2. 2D comparison
All 13 models with ScreenSpot-v2 results score lower on EgoXR-GUI overall (by 19.7–45.0 pp) and on EgoXR Explicit, our closest task match, than on v2 overall.
For space, the table shows 3 of the 13 models.
v2 scores come from the leaderboard or developers (MAI-UI, UI-Venus) for 12 models and Step-GUI for Qwen3-VL-8B; XR scores are from Table 2. Task mixes and inference protocols differ across benchmarks.
We will narrow the abstract’s overall comparison to ScreenSpot-v2 and document the sources in §4.2.
W2(ii) / Q2. Fine-tuned model
Semantic-only fine-tuning raised Semantic accuracy by about 18 pp, lowered Explicit by 4.54 pp and left Spatial and Overall statistically unchanged.
The fine-tuning is exploratory, validating synthetic data on Semantic grounding (hardest to collect and annotate, weakest for models), so the submission reported only that task; nevertheless, full results follow. We re-ran it and evaluated the checkpoint and its base, Qwen3-VL-8B, on the full benchmark under identical inference settings (a controlled comparison). These are not completely identical to Table 2’s, so the base row differs slightly; the conclusion is unaffected, as the Semantic gain is consistent (Table 4: +18.29 pp, CI [+10.18, +26.49]; here: +17.68 pp).
Base → fine-tuned (Δ pp, 95% CI): Explicit 82.67% → 78.13% (−4.54 [−6.68, −2.43]); Spatial 48.30% → 48.30% (0.00 [−5.91, +6.40]); Semantic 27.44% → 45.12% (+17.68 [+10.30, +25.45]); Overall 68.54% → 68.16% (−0.37 [−2.55, +1.78]). The modest Explicit regression is expected, as training used only Semantic instructions.
We will report all three tasks and Overall in the revised Table 4.
W3 / Q3. Existing XR benchmarks
Orienter and GUI-World belong in Table 1, but they target element detection and video QA rather than instruction-driven grounding.
Neither locates a target from an instruction or scores a click; EgoXR-GUI does both, and its instructions can depend on the physical scene (e.g., the product card for the item in the wearer’s hand). Orienter · GUI-World
We will add these comparisons to Table 1 and §2.2.2.
Code and detailed data for these results are available on request.