acceptodds
Under review as a conference paper at ICLR 2027

GazeWarp: Gaze-Guided Adaptive Warping for Bandwidth-Efficient Multimodal Interaction in XR

Abstract

Emerging wearable mixed reality (XR) systems increasingly adopt edge–cloud architectures, placing severe bandwidth constraints on the transmission of egocentric visual data to backend models. At the same time, natural XR interaction often involves ambiguous query expressions, with eye gaze serving as a practical cue to user intent. This intersection creates a critical challenge for gaze-guided multimodal interaction in XR: how to preserve the essential visual evidence needed to resolve a user's ambiguous query expression without exceeding a limited transmission budget. To study this problem, we introduce GazeRefer-GQA, a benchmark derived from GQA that combines user-view gaze cues, ambiguous query expressions, and structured reasoning annotations. We further propose GazeWarp, a gaze-guided visual enhancement framework that acts as a plug-and-play frontend before a frozen backend model. Using the user's gaze and question semantics, GazeWarp locates the intended target region and any structurally related regions, and drives a density-equalized warping process that spatially redistributes image resolution toward these regions before transmission. Across a range of transmission budgets and gaze input forms, GazeWarp generally outperforms resizing, cropping, and compression baselines under approximately matched payload budgets. Notably, despite using only a fraction of the bytes, it is comparable to the unconstrained full-resolution reference and numerically exceeds it in several settings. These gains hold across multiple backend vision-language models, and GazeWarp further generalizes strongly to additional gaze-guided datasets beyond our benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
gaze-guided visual question answering, ambiguous query expressions, gaze-guided visual enhancement, adaptive warping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6DPhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a fundamental question for XR systems: they need to transmit critical visual evidence for reasoning under physical bandwidth constraints and ambiguous query expression. Existing methods attempt to use gaze cues to assist with ambiguous query expression. However, they lack both task-matching benchmarks and methods to utilize gaze cues for visual wrapping effectively. To this end, this work proposes the GazeRefer-GQA benchmark, which consists of local and global reasoning tasks with both gaze cues and ambiguous query expressions. Beyond the benchmark, it proposes GazeWarp, a lightweight module to predict the visual evidence and a density-equalized wrapping method, leading to reduced transmission bandwidth.

Critical strengths
  • This work seek to solve a fundamental and interesting question that how to locate and transmit visual evidence under ambiguous query expression and bandwidth constraints.
  • The proposed GazeRefer-GQA benchmark have established a well-organized construction pipeline and evaluation for the community.
  • The proposed GazeWarp is plug-and-play method, and authors have conduct solid and comprehensive experiments across different backbones demonstrating its superior performance.
  • The paper have good writing and easy to follow.
Critical weaknesses
  • The cross-dataset evaluation remains limited compared to the extensive experiments on GazeRefer-GQA.
  • Lacks ablation on the isolating the contribution of region predictor and density-equalized wrapping method, respectively.
Questions for authors
  • My major concern with this work is that Gazewarp seems to be closely tied to the proposed benchmark, GazeRefer-GQA. Could Gazewarp be evaluated more extensively on other public benchmarks with gaze cues?
  • Do authors evaluate it on the real-world scenarios beyond synthetic benchmarks?
Reviewer_S86YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GazeWarp, a gaze-guided adaptive warping framework for bandwidth-efficient visual question answering. The paper also introduces a benchmark GazeRefer-GQA constructed from GQA with ambiguous query expressions and simulated gaze. The framework predicts target and related regions, then reallocates image resolution toward them using density-equalized warping. The problem is practically relevant to XR applications, but the method and experiments have notable limitations. These limitations prevent actual applications in XR, please see weaknesses. I recommend rejection as the paper does not establish either the practical feasibility of the proposed framework or substantial improvement coming from the proposed method.

Critical strengths

The bandwidth problem is nicely identified and practically relevant to XR applications. The benchmark is also meaningful contribution. The wrapping idea has novelty compared with conventional cropping methods.

Critical weaknesses

GazeWarp requires a pretrained VLM backbone for region prediction before the image is transmitted, which I suspect is very computationally heavy, and whose computational overhead and feasibility on resource-constrained XR devices are not evaluated. Although bandwidth efficiency is the central motivation, these aspects significantly limit practical application. If it runs on the XR device, it introduces substantial computation to a system that is explicitly motivated by limited on-device resources. If it runs on the backend, how can it reduce the bandwidth needed to transmit the original image? This is not a minor implementation detail. The experiments only measure transmitted bytes without demonstrating a complete pipeline.

There is also a lack of real data. The GazeRefer-GQA benchmark relies on synthetic gaze generated from ground-truth annotations, and no real human gaze evaluation is provided, so it has limited applicability to real XR. The additional VOILA-COCO evaluation also uses simulated gaze. The Gaussian perturbations are a reasonable starting point, but real gaze does not always align with the intended target, and its errors are not necessarily proportional to object size. I am therefore not convinced that simulation performance can translate to real XR performance.

Third, there is limited technical novelty over existing adaptive warping methods, and improvements over stronger cropping baselines are incremental. The paper does not clearly distinguish its contribution from existing work such as Dalal et al. (2026), which is also cited in the paper.

Overall, the paper demonstrated a narrower contribution than what is claimed. Its findings would need substantial additional experiments with real data to generalize to real XR applications.

Questions for authors

None

Reviewer_JY83OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors introduce GazeWarp, a gaze + question-guided image warping framework for bandwidth-efficient multimodal interaction in XR. The method predicts the gaze-indicated target and a query-dependent related region, redistributing image resolution toward these regions before transmission to a frozen backend VLM. The authors also introduce GazeRefer-GQA derived from the popoular GQA benchmark, augmented with ambiguous questions and simulated gaze cues, covering Local and Global reasoning.

Critical strengths
  • Clear problem motivation and corresponding contribution. The paper jointly addresses query disambiguation and bandwidth-constrained visual understanding -- a practically relevant combination for wearable assistants & timely contribution with emerging XR devices specifically.

  • Preserving target detail and broader context. Unlike target-centered cropping, the proposed method allocates resolution to both the gazed object and a related region, while retaining the surrounding scene for relational reasoning.

  • Structured benchmark and modular design. The local & global split allows researchers to distinguish target grounding from relational reasoning, and the trained frontend can be used with different frozen backend VLLM models without fine-tuning.

  • Empirical analysis provides interesting insights, and specifically promising accuracy improvement with gaze input.

Critical weaknesses
  • Synthetic nature of the benchmark & limtied XR realism -- all gaze input is synthetically generated from object bounding boxes with target-relative Gaussian jitter, rather than measured eye tracking from actual devices. This holds onto an important assumption that user eye gazes always fall onto target objects or related regions (to the query), whereas in reality there might exist negative or noisy gaze input unrelated to verbalized objects.

  • Incomplete validation of benchmark gaze dependency -- rewriting questions into pronoun-based forms does not necessarily make gaze essential e.g. when there is only one prominent object in the scene, etc. The limited empirical gains with gaze input allude to this potential issue as well - it'd be good to clarify.

  • Approximately matched budgets and small numerical gains in general. Tighter byte matching and uncertainty estimates would strengthen these comparisons.

Questions for authors

Please see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.