acceptodds
Under review as a conference paper at ICLR 2027

EgoXR-GUI: Benchmarking GUI Grounding in Physical–Digital Extended Reality

Abstract

The development of multimodal large language models has significantly advanced intelligent graphical user interface (GUI) agents. However, current research and benchmarking mainly focus on traditional platforms, like computer desktop and mobile interfaces, without considering the rapidly developing field of extended reality (XR). Unlike traditional platforms, XR requires models to understand the vital link between digital overlays and complex physical contexts. Moreover, unique platform artifacts and real-world egocentric interactions introduce additional challenges, such as perspective distortion, hand occlusion, and cluttered backgrounds. To this end, we present EgoXR-GUI, the first XR-specific GUI grounding benchmark, spanning direct, spatial, and semantic grounding tasks, and featuring over 1,000 high-quality annotated samples collected across diverse real-world scenarios. Our results show that state-of-the-art models perform significantly worse on these tasks than on conventional 2D grounding, revealing a critical gap in their ability to reason within hybrid physical-digital environments.

Primary area:
datasets and benchmarks
Keywords:
GUI Grounding, Extended Reality, Egocentric Vision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KkEoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EgoXR-GUI, a benchmark for GUI grounding in physical–digital extended reality (XR) environments. It contains 1,068 annotated samples across 47 applications and interfaces, covering three tasks: Explicit, Spatial, and Semantic Grounding. The authors evaluate 28 existing models and explore synthetic-data fine-tuning to improve grounding performance.

Critical strengths
  1. The benchmark contributes valuable long-tail GUI grounding data by introducing real-world XR scenarios.
  2. The evaluation covers 28 models with different architectures and training paradigms, providing a useful baseline for future XR GUI grounding research.
Critical weaknesses
  1. Limited analysis of physical backgrounds. The paper repeatedly highlights complex physical backgrounds as a key challenge in XR grounding, but does not systematically examine their impact. Controlled experiments with varying background complexity would help clarify how much they contribute to grounding difficulty.
  2. Incomplete experimental results. The Abstract claims that existing models perform worse on EgoXR-GUI than on conventional 2D grounding tasks, but direct experimental results are not provided. Moreover, Table 4 only reports Semantic Grounding results for the fine-tuned model, leaving its performance on Explicit and Spatial Grounding, as well as the full benchmark, unclear.
  3. Limited comparison with existing XR benchmarks. Table 1 (Lines 108–117) only includes conventional Desktop, Web, and Mobile GUI benchmarks, omitting related XR works such as Orienter and GUI-World. Although discussed in Section 2.2.2 and Figure 2(a), these works deserve a more systematic comparison on table.
Questions for authors
  1. How does physical background complexity affect grounding performance? Could the authors provide controlled experiments to examine this?
  2. Can the authors provide a direct comparison with conventional 2D grounding and report the fine-tuned model's performance on all three task categories?
  3. Could the authors include existing XR benchmarks in Table 1 to better clarify the differences from prior work?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer KkEo

We thank the reviewer for recognizing our real-world XR data and 28-model baseline, and address each concern in order.

W1 / Q1. Background intervention

Our controlled experiment shows that, for these models and conditions, weakening the scene around the UI changes Explicit accuracy only slightly, so complex backgrounds are not a major interference for high-accuracy models.

Explicit instructions name the target directly, isolating scene interference. On all 727 (206 screenshots), we fixed screenshot, instruction, box and inference settings and altered only pixels outside a mask preserving the UI, target boxes and translucent-panel content: Gaussian blur (σ = 4, 8, 16, 30 px) or uniform gray (RGB 128). Clicks inside the box count as correct.

Model (%)Uneditedσ=4σ=8σ=16σ=30Gray
Qwen3-VL-8B84.0484.8784.7384.4684.5984.46
MAI-UI-8B88.4589.1388.5888.1788.7288.45
UI-Venus-1.5-8B88.0389.0089.1389.2789.4189.68

Unedited predictions matched Table 2 hit for hit for Qwen3-VL-8B and UI-Venus-1.5-8B; MAI-UI-8B differed by two hits (643 vs. 641/727), so changes use the unedited run. Changes ranged from −0.28 to +1.65 pp, and all 15 simultaneous 95% intervals (screenshot-level paired bootstrap, 50,000 draws) included zero; the largest was UI-Venus-1.5-8B with gray, +1.65 pp [−0.54, +3.85].

By design, Spatial and Semantic instructions refer to physical objects or positions, so the scene matters more there.

We will add this experiment with more baselines to the paper and narrow the claim that visual clutter impairs Explicit grounding.

W2(i) / Q2. 2D comparison

All 13 models with ScreenSpot-v2 results score lower on EgoXR-GUI overall (by 19.7–45.0 pp) and on EgoXR Explicit, our closest task match, than on v2 overall.

For space, the table shows 3 of the 13 models.

Model (%)ScreenSpot-v2XR ExplicitSpatialSemanticOverall
Qwen3-VL-8B94.4084.0451.1423.1769.29
MAI-UI-8B95.2088.1764.7730.4975.47
UI-Venus-1.5-8B95.9088.0369.8928.6675.94

v2 scores come from the leaderboard or developers (MAI-UI, UI-Venus) for 12 models and Step-GUI for Qwen3-VL-8B; XR scores are from Table 2. Task mixes and inference protocols differ across benchmarks.

We will narrow the abstract’s overall comparison to ScreenSpot-v2 and document the sources in §4.2.

W2(ii) / Q2. Fine-tuned model

Semantic-only fine-tuning raised Semantic accuracy by about 18 pp, lowered Explicit by 4.54 pp and left Spatial and Overall statistically unchanged.

The fine-tuning is exploratory, validating synthetic data on Semantic grounding (hardest to collect and annotate, weakest for models), so the submission reported only that task; nevertheless, full results follow. We re-ran it and evaluated the checkpoint and its base, Qwen3-VL-8B, on the full benchmark under identical inference settings (a controlled comparison). These are not completely identical to Table 2’s, so the base row differs slightly; the conclusion is unaffected, as the Semantic gain is consistent (Table 4: +18.29 pp, CI [+10.18, +26.49]; here: +17.68 pp).

Base → fine-tuned (Δ pp, 95% CI): Explicit 82.67% → 78.13% (−4.54 [−6.68, −2.43]); Spatial 48.30% → 48.30% (0.00 [−5.91, +6.40]); Semantic 27.44% → 45.12% (+17.68 [+10.30, +25.45]); Overall 68.54% → 68.16% (−0.37 [−2.55, +1.78]). The modest Explicit regression is expected, as training used only Semantic instructions.

We will report all three tasks and Overall in the revised Table 4.

W3 / Q3. Existing XR benchmarks

Orienter and GUI-World belong in Table 1, but they target element detection and video QA rather than instruction-driven grounding.

BenchmarkPlatformInputScaleTaskOutputMetricExpl./Spat./Sem.
OrienterVR (Steam VR apps)Right-eye views1,552 views, 100 apps, 4,470 elementsDetect interactable elements, label semantics via app contextBoxes, labelsPrecision, recall, F1–/–/–
GUI-WorldCross-platform incl. XRVideos, keyframes, questions12,379 videos (393 XR)Describe action sequences or predict next actionTextMCQ accuracy; LLM scoring–/–/–
EgoXR-GUIXR (Vision Pro), real scenesScreenshot + instruction1,068 instructions, 233 screenshots, 47 appsLocate the instructed controlClick pointPoint-in-box accuracy✓/✓/✓

Neither locates a target from an instruction or scores a click; EgoXR-GUI does both, and its instructions can depend on the physical scene (e.g., the product card for the item in the wearer’s hand). Orienter · GUI-World

We will add these comparisons to Table 1 and §2.2.2.

Code and detailed data for these results are available on request.

Thank you for the additional experiments. The responses to W2 and W3 have addressed my concerns.

However, W1 remains partially addressed. The experiment only examines Explicit Grounding, leaving the impact of physical background complexity on Spatial and Semantic Grounding unexplored.

A complexity-stratified analysis across all three grounding task types, if possible, would help clarify the impact of physical backgrounds and better support the paper's motivation.

Thank you for confirming that W2 and W3 are addressed. We agree that the effect of physical backgrounds on Spatial and Semantic grounding is central to our motivation, and appreciate the suggestion to examine all task types. In an exploratory complexity-stratified analysis, all three 8B baselines score lower on Spatial and Semantic grounding in higher-clutter scenes. Removing the background also lowers Spatial and Semantic accuracy for all four models tested, with the largest Semantic drop for Gemini.

Spatial and Semantic instructions rely on physical context: spatial relations link physical referents to UI targets, and physical objects or their properties determine the intended option. Clutter can make this inference harder, while blurring or masking removes the context itself. We examine the two effects separately. Qwen, MAI and Venus denote Qwen3-VL-8B, MAI-UI-8B and UI-Venus-1.5-8B; all values are percentages and Δ is in percentage points.

1. Complexity-stratified analysis

We used GPT-6-Astra to rate task-irrelevant physical clutter on a 1–5 scale without access to model predictions or correctness, and averaged scores within each screenshot and task type. Using reviewed task labels, we compare low (≤2) with higher (>2) clutter; accuracy is instruction-weighted.

Task (n low / higher)ModelLowHigherΔ
Spatial (29 / 121)Qwen55.1740.50−14.68
SpatialMAI58.6253.72−4.90
SpatialVenus62.0752.89−9.18
Semantic (8 / 113)Qwen50.0017.70−32.30
SemanticMAI37.5028.32−9.18
SemanticVenus50.0028.32−21.68

Higher clutter is associated with 4.90–14.68 pp lower Spatial and 9.18–32.30 pp lower Semantic accuracy for all three models.

Place–activity pairs illustrate this. For Spatial, living room × exercise centers on curtains, radiators and a treadmill, whereas living room × daily routine shows wider room views with furniture, decorations and dispersed windows (median background edge density 0.230% vs. 0.949%). For Semantic, restaurant × ordering food usually shows an isolated food item, whereas supermarket × shopping shows dense shelves of similar products (2.166% vs. 7.115%).

ModelExercise (n=20)Daily routine (n=29)ΔRestaurant (n=9)Supermarket (n=15)Δ
Qwen70.0031.03−38.9733.3313.33−20.00
MAI85.0055.17−29.8355.566.67−48.89
Venus75.0072.41−2.5933.3320.00−13.33

All three models also score lower in the more complex pair: by 2.59–38.97 pp on Spatial and 13.33–48.89 pp on Semantic.

These are exploratory associations, not causal evidence. The low-clutter Semantic group has only eight instructions, tests on the continuous score were not significant, other scoring dimensions gave mixed trends, and the paired subsets also differ in task and UI content (and, for Semantic, in place).

2. Background removal

We compare original images, Gaussian blur (σ = 30 px) and uniform gray (RGB 128), preserving the UI; instructions, target boxes, labels and inference settings are fixed.

ModelTaskOriginal (= Table 2)σ30Δσ30GrayΔGray
QwenSpatial51.1438.42−12.7236.16−14.98
QwenSemantic23.1722.56−0.6122.56−0.61
MAISpatial64.7745.76−19.0145.76−19.01
MAISemantic30.4918.90−11.5921.34−9.15
VenusSpatial69.8949.15−20.7446.89−23.00
VenusSemantic28.6620.12−8.5423.17−5.49

Because these baselines start from low Semantic accuracy, we added Gemini 3.1 Flash, a newer release than the Gemini 3 Flash Preview in Table 2. Each intervention is compared with Gemini 3.1 Flash’s own original-image run; the Table 2 results are unchanged.

Gemini 3.1 FlashOriginalσ30Δσ30GrayΔGray
Explicit86.3889.13+2.7589.41+3.03
Spatial67.2359.89−7.3453.11−14.12
Semantic65.8541.46−24.3934.76−31.10

Gray lowers Spatial and Semantic accuracy for all four models, by 5.49–31.10 pp on Semantic. Gemini shows the largest Semantic drop (31.10 pp), with a 14.12 pp Spatial drop and a small Explicit gain, underscoring the role of physical context in background-dependent reasoning.

Spot-checks explain the remaining accuracy: instruction or UI text, small candidate sets and selection biases offer alternative cues. In Semantic #1003, the selected “iced” option remains after the drink label is removed; in Spatial #695, the timer windows keep their left–right arrangement after the pots disappear. Segmentation also leaves some physical fragments around protected or translucent UI, so gray does not remove every usable cue.

In the revision, we will add both analyses. For the camera-ready version, we will also collect matched simple and complex backgrounds for the same tasks, preserving the relevant physical cues and UI, to isolate the effect of complexity.

Thank you again for this constructive suggestion, which has helped us clarify how physical backgrounds affect each grounding task.

Thank you for the thoughtful and detailed response, which addresses most of my concerns.

However, there is still room for improvement. While EgoXR-GUI introduces a valuable benchmark for an important XR GUI grounding problem, it does not yet fully explain what makes this setting particularly challenging. The dataset is also relatively limited in scale. Additionally, polygon-based annotations may be more appropriate for XR interfaces, and the visual quality and clarity of the figures could be further improved.

Therefore, I decide to maintain my current rating, although my overall assessment remains borderline between Weak Reject and Weak Accept.

Reviewer_vPHWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoXR-GUI, a GUI grounding benchmark for VR/XR GUI environments. The grounding tasks cover explicit target naming (explicit grounding), intention specification (semantic grounding), and positional description (spatial grounding). The data is further split by icon vs. text grounding and by environment/scenario type. A data collection pipeline with multiple layers of human collection and validation is implemented. A wide variety of models are tested, and some show reasonably good but below-human results. Finally, a training experiment with additional synthetic data is conducted, showing mixed results.

Critical strengths

(+) A fairly solid introduction of a new dataset for the research community. It appears to have a good quality control protocol for human annotators and validators. I expect it could be useful for other researchers who want to explore VR/XR GUI agents, if released.

(+) Novelty. The claim that this is the first XR grounding benchmark/dataset appears to be mostly accurate based on my search.

(+) Easy to understand and informative, with a nice inclusion of an error analysis.

Critical weaknesses

(-) Lacks statistical analysis.

  • The composition of the introduced dataset should be stated more explicitly, especially (1) the size of each split, such as environment, scenario, and grounding instruction type, and (2) whether there is significant imbalance among these subcategories; if so, a report of the balanced average accuracy of all models would be helpful.
  • This raises a question regarding the stability of the reported results, since there are many categories over a relatively small dataset (~1k). Repeated trials and reports of statistical significance would be helpful.

(-) The experiment and analysis related to training with synthetic data could be improved. Currently, the experiment uses a small dataset from a single source for synthesis and a single training method, and the results are mixed. It does not seem that any conclusion can be drawn from the current experiment. This is understandable, though, since it is not the focus of the paper.

  • Additionally, no post-training results on the explicit and spatial evaluations are reported. These should be tested and reported as a regression guard.

(-) Minor:

  • More details on the recruitment, demographics, IRB approval (if needed), and compensation of the data collectors and annotators would be helpful. Importantly, it should be stated whether some or all of the humans involved are also authors.
  • The categorization of grounding tasks (e.g., in Section 3.1) sounds similar to prior work in GUI grounding and is worth some citations. See, e.g., UGround, Section 2.2.
  • If time and budget allow, broader coverage of closed-source models in the main results table would be helpful. If not, this should be noted.
Questions for authors
  • Clarification: what would the results be if the trained model were tested on the spatial and explicit splits of the dataset?
  • Clarification: what are the sizes of the text/icon splits, the explicit/spatial/semantic splits, the scenario splits, and the environment splits? If there is an imbalance, is it a cause for concern, or is it acceptable?
  • Future work: the results seem to suggest that targeted training on flat GUI grounding helps improve model performance on VR/XR GUI grounding. As a future direction, would the reverse also work, especially as a way to diversify GUI datasets?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer vPHW

We thank the reviewer for recognizing the benchmark’s novelty, quality control and error analysis, and address each concern below.

W2(ii) / Q1. Explicit/Spatial after fine-tuning

Semantic-only fine-tuning raised Semantic accuracy by about 18 pp, lowered Explicit by 4.54 pp and left Spatial and Overall statistically unchanged.

The fine-tuning is exploratory, validating synthetic data rather than maximizing overall accuracy. We thus synthesized only Semantic data, the task hardest to collect and annotate and weakest for models, and reported only that task. Nevertheless, we report all tasks here.

We re-ran Semantic-only fine-tuning and evaluated the checkpoint and its base, Qwen3-VL-8B, on all 1,068 real instructions under identical inference settings (a controlled comparison). These are not completely identical to Table 2’s, so the base row differs slightly; the conclusion is unaffected, as the Semantic gain is consistent (+18.29 pp in Table 4, +17.68 pp here).

TaskBaseFine-tunedΔ (pp)95% CI
Explicit82.67% (601/727)78.13% (568/727)−4.54[−6.68, −2.43]
Spatial48.30% (85/176)48.30% (85/176)0.00[−5.91, +6.40]
Semantic27.44% (45/164)45.12% (74/164)+17.68[+10.30, +25.45]
Overall68.54% (732/1,068)68.16% (728/1,068)−0.37[−2.55, +1.78]

The Semantic gain reproduces Table 4 (38 to 68 of 164 hits, +18.29 pp, CI [+10.18, +26.49]). The modest Explicit regression is expected, as training used only Semantic instructions.

We will report all three tasks in Table 4, limit the training claim to the Semantic gain and document both inference configurations in Appendix E.

W1(i) / Q2. Split sizes

The composition by task, target type, scenario and environment is below.

The 1,068 instructions on 233 screenshots, each counted once:

TaskTextIconTotal
Explicit456271727
Spatial7899177
Semantic12935164
Overall6634051,068

Scenarios: Home life 375, Consumption & Entertainment 319, Work & Study 216, Sport & Health 76, Outdoor trip 44, Design & Creation 38.

Environments (Fig. 2(b) labels): living room 218, supermarket 182, kitchen 157, restaurant 154, library 114, office 65, meeting room 52, street 42, playground 32, studio 30, shopping mall 22. Two entries take their screenshots’ shared living-room label.

We will recompute Spatial on all 177 instructions (one was classified as Spatial after the original 176-instruction evaluation), add these counts, update the Fig. 2(b) scenario percentages and correct the environment count to 11 in §3.2 and Appendix A.4.

W1(ii) / Q2. Balanced accuracy

The splits are imbalanced, but under task-balanced accuracy the top-ranked model is unchanged and only some adjacent ranks swap.

Participants chose settings where they naturally use XR, so Overall weights common settings more. Weighting the three tasks equally across all 28 Table 2 entries keeps UI-Venus-1.5-8B first (62.19% balanced, 75.94% overall); Holo1.5-72B (61.88%) moves ahead of MAI-UI-8B (61.14%). With 35 icon targets among 164 real Semantic instructions, one icon prediction shifts that subgroup by 2.86 pp. We will report both metrics.

W2(i) / Q2. Verified synthetic set

A manually verified synthetic set further supports that GUI-specific models perform poorly on Semantic grounding.

From samples used in fine-tuning, we built a screened set of 500 synthetic Semantic instructions (351 source images), keeping only items with a manually verified instruction and box. CIs use 10,000 bootstrap draws grouped by source image.

ModelReal Semantic (Table 2)Verified synthetic95% CI
MAI-UI-8B30.49%57.2%[52.6%, 61.7%]
Qwen3-VL-8B23.17%42.8%[38.5%, 47.1%]
UI-Venus-1.5-8B28.66%41.0%[36.7%, 45.2%]

All three scored higher than on EgoXR-GUI, yet the best still erred on 42.8%, supporting the conclusion and motivating synthetic fine-tuning.

Q3. XR data for flat GUI grounding

XR GUI data may also benefit flat GUI grounding, which we have not yet tested.

XR screenshots add viewpoint, lighting and window-placement variation absent from flat datasets. We will test whether adding XR GUI data to flat GUI training sets, with the same backbone and budget, improves flat benchmarks such as ScreenSpot-v2 and ScreenSpot-Pro.

Minor (i). Eight volunteers collected recordings, two author-team members annotated them, and three non-annotators did the human evaluation; all gave informed consent. We will clarify recruitment, available demographics, compensation and ethics-review status in §3.3 and Appendix A.

Minor (ii). We will relate the three task types in §3.1 to earlier GUI referring expressions (appearance, function, position; UGround, §2.2).

Minor (iii). API budget and evaluation workload limited the closed-source scope; we will extend it in a later version.

Code and detailed data for these results are available on request.

Reviewer_vPHWOpenReview ↗

Thank you

Thank you to the authors for the reply. I have one lingering concern

  • The authors made no mentions of adding repeated trials and confidence intervals of the main results (table 2). I assume this can be due to time constraints of the rebuttal. I still believe this is worth adding and would make the result of the paper more rigorous.

Otherwise, if important statistics and claims made in this reply can be added into the paper / appendix, I have no further concerns.

AuthorsOpenReview ↗

Reply to Follow-up: Repeated Trials and Bootstrap CIs for Table 2

Thank you for your careful follow-up and for highlighting this remaining concern. We agree that repeated trials and confidence intervals for the main results would make the paper more rigorous, and we ran this analysis for two models in the remaining rebuttal time.

Repeated trials. We ran three independent full-benchmark evaluations of Qwen3-VL-8B and UI-Venus-1.5-8B, reloading the model and regenerating all predictions in each trial.

SettingConfiguration
Trials3 per model, each on all 1,068 instructions
Seeds42, 123, 2026
DecodingGreedy (do_sample=False, i.e., temperature 0), max 256 new tokens, batch size 1
Prompts, preprocessing, scoringOriginal model-specific implementations
Hardware and software1× NVIDIA RTX 5090; PyTorch 2.8.0+cu128, Transformers 4.57.0, FlashAttention 2.8.3.post2

For both models, the run-to-run standard deviation is 0.00 pp for every task and Overall. Under greedy decoding, the model’s outputs for the same input differ very little across runs, and point-in-box scoring is not sensitive to such small differences, so the accuracies were identical across the three trials.

Confidence intervals. A larger source of uncertainty is the finite size of the benchmark. We drew 10,000 screenshot-level bootstrap resamples, keeping all instructions from a sampled screenshot together, and report 95% percentile intervals.

ModelTaskTable 2Repeated runs95% CI
Qwen3-VL-8BExplicit84.04%83.49%[80.77%, 86.16%]
Qwen3-VL-8BSpatial51.14%51.70%[43.62%, 60.00%]
Qwen3-VL-8BSemantic23.17%22.56%[15.66%, 29.89%]
Qwen3-VL-8BOverall69.29%68.91%[65.53%, 72.20%]
UI-Venus-1.5-8BExplicit88.03%87.90%[85.28%, 90.33%]
UI-Venus-1.5-8BSpatial69.89%68.18%[61.39%, 74.61%]
UI-Venus-1.5-8BSemantic28.66%28.05%[20.86%, 35.58%]
UI-Venus-1.5-8BOverall75.94%75.47%[72.65%, 78.22%]

Differences from Table 2. Due to a change in computing resources, these runs used an RTX 5090 and a newer FlashAttention version than the original runs, which changed a small number of predictions. Overall accuracy differs from Table 2 by 0.37 pp for Qwen3-VL-8B and 0.47 pp for UI-Venus-1.5-8B, the largest task-level difference is 1.71 pp (UI-Venus-1.5-8B, Spatial), and every Table 2 value lies within the corresponding 95% CI.

Revision. In the camera-ready version, we will extend repeated evaluations to all models in Table 2 and report the mean accuracy, run-to-run standard deviation and 95% bootstrap CI for each task and Overall, together with the trial counts, inference configurations, software versions and resampling procedure. We will also add the statistics and claims from our previous response to the main paper and appendix, including the task-wise fine-tuning results, split counts and the evaluation on the verified synthetic Semantic set.

Thank you again for helping us strengthen the empirical evaluation.

Reviewer_x1pSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EgoXR-GUI, the first GUI grounding benchmark designed for physical–digital hybrid XR environments. The benchmark comprises 1,068 samples collected using Apple Vision Pro across 47 applications and categorizes grounding tasks into three types based on their underlying evidence: Explicit, Spatial, and Semantic. The authors evaluate various models and further explore two potential improvements: a Task Router inference framework and fine-tuning with synthetic data, which improves Semantic task accuracy from 23.17% to 41.46%.

Critical strengths

1. Novel problem formulation with strong motivation. The paper addresses the interaction between virtual interfaces and physical environments in XR, including hand occlusion, perspective distortion, multiple windows, and ambiguous references to physical objects. This benchmark provides a valuable foundation for future research on XR agents.

2. Broad evaluation with informative findings. The evaluation covers 28 models across five categories, reports text and icon grounding separately, and includes a human upper bound (99.44% overall). The four-type Semantic error analysis, attention heatmaps, and Thinking-variant ablation (Table 8) offer actionable insights for future work.

3. Multiple supplementary validations and ethical documentation. The paper reports annotation audits (Table 5, 99–100% pass rates, though conducted by the authors themselves), a human performance evaluation, a Pico 4 Ultra cross-device diagnostic (18 pairs selected from failure cases, so it does not estimate overall cross-device accuracy), a gaze-input experiment (a negative result: overall accuracy drops from 75.47% to 67.60%/71.54%), Chinese-instruction results (with an unexplained Gemini 3 drop from 57.30% to 31.46%), a 96% pass rate on a 300-sample audit of the synthetic data, and discussions of privacy and licensing.

Critical weaknesses

W1. Typos, unexplained anomalies, and an under-specified evaluation protocol. The Table 1 caption refers to the benchmark as "Ego-GUI" (Line 108), and Fig. 2 is labeled "Ego-GUI (Ours)" (Line 223), inconsistent with the name EgoXR-GUI used elsewhere. The Table 9 caption claims 28 models but lists 20. Section 4.1 states that all models use "official prompt templates", and Appendix E.1 lists many of them. But it gives none for Gemini 3 or the Qwen family, and it does not say how coordinate formats or image resolution were handled for high-resolution XR screenshots. Evaluation fairness therefore cannot be verified.

W2. Insufficient sample size and no statistical analysis. Back-calculating from Table 2, the Semantic subset has about 164 samples (23.17% = 38/164), of which only 35 are icon targets (20.00% = 7/35; 17.14% = 6/35). The icon "decrease" after fine-tuning in Table 4 therefore corresponds to a single sample, yet the paper reports it as a drop. In Table 3, Semantic-icon differences between frameworks and MAI-UI-8B correspond to only 4–8 samples. The Spatial subset has about 176 samples (77 text, 99 icon), so 1–3 point gaps between models amount to a few samples. No confidence intervals, bootstrap estimates, or significance tests are reported for any accuracy result. This matters most for the Router claim: it has the highest Overall among the compared frameworks (75.93), but it exceeds MAI-UI-8B alone (75.47) by 0.46 points, roughly 5 samples, which cannot be distinguished from noise.

W3. The "RL outperforms SFT" conclusion is confounded. The gap between groups in Table 2 is large, but the groups are not comparable. The SFT group includes older models (OS-Atlas, UGround, CogAgent) alongside OpenCUA, while the RL group consists of more recent models. The paper reports neither base models nor training data, so base model, data scale, and release date are all uncontrolled. The groups also overlap: OpenCUA-32B (SFT, 62.64) outperforms several RL models, including UI-TARS-1.5-7B (48.50), Tianxi-Action-7B (56.74), and Holo2-4B/8B (56.93/61.42). The explanation offered (SFT overfitting versus RL exploration) is untested, and a controlled comparison on the same base and data is missing. Some results also look anomalous. OpenCUA-72B scores below both the 32B and 7B variants in English (47.94 vs 62.64 and 53.28), and its Chinese-instruction Overall collapses to 10.11 (Table 9), which suggests a prompt or output-format failure rather than a capability difference.

W4. Synthetic-data pipeline. Bounding-box labels for the synthetic data come from UI-Venus-1.5-8B, which is itself an evaluated baseline (the highest Overall in Table 2, 75.94) and was chosen as the labeler partly for its benchmark performance. Fine-tuning therefore distills this model's labels, and label quality is capped by its roughly 88% Explicit accuracy. The human audit covers only 300 of 1,500 annotations (96% pass rate), with no description of how the 300 were sampled. The overlays are also generated by an image-generation model, limited to shopping and service apps, and the gap between this synthetic UI and real Vision Pro UIs is neither quantified nor discussed.

Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Reply to Reviewer x1pS

We thank the reviewer for recognizing our XR formulation, 28-model evaluation and validations.

W1. Protocol and corrections

The naming and caption issues are typos, and Gemini’s Chinese drop is sensitive to coordinate order.

“Official templates” refers to the ScreenSpot-Pro evaluation code. Gemini 3 Flash Preview received native-resolution PNGs; most of its boxes used integer 0–1000 coordinates, which we normalized and read x-first. After patch-aligned resizing, Qwen2.5-VL returned points in resized-image pixels and Qwen3-VL on a 0–1000 grid. A point or box center mapped onto or inside the target box scored 1; misses and unparseable outputs, 0.

Gemini fell from 57.30% (English) to 31.46% (Chinese), with only 1 vs. 6 unparseable responses. Swapping x and y in all saved predictions lowers English to 35.11% and raises Chinese to 58.52%, so the gap is sensitive to coordinate order and limits what Table 9 shows about language understanding. Table 9 also covers only 20 of the 28 Table 2 entries in Chinese, which its caption does not reflect.

We will evaluate the remaining eight entries in Chinese and update the Table 9 caption, use “EgoXR-GUI” in Table 1 and Fig. 2, and post the exact Gemini and Qwen prompts and scoring rule in a follow-up and Appendix E.1.

W2. Statistical uncertainty

Paired bootstrap CIs show that the Router’s and fine-tuning’s Semantic gains are significant, unlike the Router’s overall edge over MAI-UI-8B and the one-sample icon change.

Over 10,000 paired draws of the 233 screenshots, Router − MAI-UI-8B is Overall +0.47 pp [−2.09, +2.95], Explicit −5.09 [−7.60, −2.66], Spatial +2.27 [−4.22, +8.84] and Semantic +23.78 [+14.71, +32.74] (95% CIs): the Router trades an Explicit loss for a Semantic gain. Fine-tuning gains +18.29 pp [+10.18, +26.49] on all 164 Semantic instructions; one fewer icon hit (7/35 to 6/35, −2.86 pp [−16.22, +8.33]) does not show an icon regression.

We will report the Router trade-off in Table 3, revise the icon wording and add counts and CIs to Tables 3 and 4.

W3. RL vs. SFT; OpenCUA-72B

Although Table 2 is not a controlled comparison, adding an RL stage improves grounding over SFT alone in prior backbone-fixed ablations, and OpenCUA-72B’s low scores partly reflect format failures.

Table 2 mixes backbones, data and release dates; OpenCUA-32B (SFT, 62.64%) beats several RL models. Examples on ScreenSpot-Pro: INREACT, 20.7% with second-stage RL vs. 16.8% with SFT (Qwen2-VL-2B); UI-Ins, 52.2% with SFT+RL vs. 37.1% with SFT (Qwen2.5-VL-7B); UI-Venus-1.5-8B, 63.1% after mid-training vs. 70.0% after offline RL. INREACT · UI-Ins · UI-Venus

OpenCUA-72B gave 157 unparseable answers in English (14.7%) vs. 389 in Chinese (36.4%), and parseable misses rose from 399 to 571; both drove its fall from 47.94% to 10.11%.

In §4.2 we will frame the RL–SFT gap as a trend consistent with these ablations, drop the untested overfitting/exploration account and note OpenCUA-72B’s format sensitivity.

W4. Synthetic labels

The pipeline does not distill UI-Venus’s Semantic ability: it sees only Explicit instructions, and the fine-tuned model surpasses it on real Semantic tasks. Label errors can propagate, yet the gain shows the synthetic data is effective.

For an EPIC-KITCHENS frame with scissors in hand, we render a shopping panel with scissors and distractors, and Qwen3.6-Plus writes paired Explicit (“the scissors”) and Semantic (“the tool in hand”) instructions. UI-Venus-1.5-8B (88.03% Explicit accuracy) sees only the image and Explicit instruction; its predicted box center becomes the Semantic target. Two reviewers checked 300 random targets; instruction validity and box accuracy each passed on 288 (96%).

CLIP ViT-B/32 mean cosine (VP = real Vision Pro captures):

ComparisonCos.BaselineCos.Δ [95% CI]
Synthetic vs. VP (500 frames)0.6955Original EPIC vs. VP0.6458+0.0496 [0.0393, 0.0621]
Synthetic vs. VP (food catalogs, language-matched)0.7592Synthetic vs. 2D0.7110+0.0482 [0.0351, 0.0633]
Synthetic vs. VP (English catalogs)0.7032VP vs. VP across apps0.7334−0.0302 [−0.0427, −0.0181]

Synthesis moves images toward VP (closer to VP than to 2D), though real VP catalogs across apps stay more similar to each other.

Of 1,500 targets from 500 frames, 1,498 were usable. The exploratory fine-tuning (testing whether synthetic data helps, not maximizing overall accuracy) raised Qwen3-VL-8B’s real Semantic accuracy from 23.17% to 41.46% (Table 4; 45.12% re-run), far above UI-Venus-1.5-8B’s own 28.66%. A student surpassing its labeler is not merely distilling it.

We will add the audit’s sampling procedure and this similarity analysis to Appendix E.3.

Code and detailed data for these results are available on request.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.