acceptodds
Under review as a conference paper at ICLR 2027

Understanding What You Point At: Spatial Guidance for Reference Disambiguation in Egocentric Video

Abstract

Pointing gestures link linguistic references to physical objects, providing essential cues for egocentric AI assistants to interpret user intent and interact naturally. However, current multimodal large language models (MLLMs) struggle to distinguish intended targets from distractors in spatially complex and dynamic scenes. Existing benchmarks provide limited systematic coverage of these ambiguities. We introduce PointConfusion, to our knowledge the first egocentric video understanding benchmark designed around a taxonomy of spatial and dynamic pointing ambiguities. Its 5,000 synthetic videos enable systematic evaluation of five ambiguity categories through a unified target-selection task. Our method, PointMod, learns spatial reference heatmaps from hand geometry and uses them to modulate visual features, explicitly incorporating pointing information into the model’s visual representation. Extensive evaluations on PointConfusion demonstrate that PointMod outperforms a range of advanced MLLMs and achieves state-of-the-art performance. PointMod-8B reaches 88.33% target-selection accuracy, improving upon task-fine-tuned InternVL3.5-8B by 8.13 percentage points. The dataset and code will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Vision, Gesture Grounding, MLLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iUBmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PointConfusion, an egocentric video benchmark for controlled pointing gesture grounding in multiple choice question answering format. The dataset controls 5 pointing ambiguity categories: depth overlap, ray ambiguity, saliency distraction, temporal confusion, and viewpoint drift, 15,000 total questions from 5,000 synthetic videos, where 1,500 questions are used for test.

The authors also propose PointMod method, which predicts a heatmap from reconstructed 107-dimension hand information, supervised by target bounding boxes. This heatmap is used to modulate visual tokens before putting to LLM. On PointConfusion, an experiment with InternVL3.5-8B showed that PointMod improves 8.3% compared with standard finetuning.

Critical strengths
  1. PointConfusion introduces a structured types of pointing ambiguities, and constructs videos with controlled conditions. This category-wise evaluation helps identifying which types of ambituity is challenging for existing MLLMs.

  2. PointMod provides a simple but effective way to inject pointing geature information into visual representations. It's relative gain compared with RGB finetuning suggests that explicit spatial guidance is useful.

Critical weaknesses
  1. The connection to real-world pointing gesture understandig is unclear. Five conditions are motivated by prior work, but the paper does not show how representative they are of the challenges in real world pointing videos. For example, motion blur or partial hand occlusion can make hand reconstruction unreliable, potentially affecting PointMod's spatial guidance. While the controlled synthetic setup is useful for isolating specific ambiguities, PointMod's performance on real-world pointing gestures remains unclear. A small real-world evaluation or further discussion of these limitations would strengthen the practical relevance of the work.

  2. Generated gesture videos are hard to assess from frame visualizations. For example, in Figure 2 and S3, Temporal confusion example made a bit difficult to follow which of the two objects are referred.

Questions for authors
  • Could the authors explain on the selection of the five ambiguity categories and their relevance to real-world pointing scenarios? In particular, have the authors considered the effects of motion blur or hand occlusion on PointMod, given its reliance on reconstructed hand features?
  • Could the authors provide video clips illustrating the five ambiguity types?
  • Could the authors compare performance using on a pointing-hold frame against the full video? it might help
  • Could the authors compare PointMod with another way of using the same learned heatmap, such as overlaying it on the video frames? This would help separating the benefits from heatmap from visual modulation.
  • Is it possible to evaluate PointMod on a small set of real-world pointing videos?
5 comments
AuthorsOpenReview ↗

Response to Reviewer iUBm (1/2)

Thank you for recognizing the diagnostic value of PointConfusion and the spatial-guidance design of PointMod. Following your suggestions, we have strengthened the literature grounding for the five ambiguity categories, added robustness evaluations and video demonstrations, and incorporated the experiments and explanations into the revised manuscript.

1. Selection of the five ambiguity categories and their real-world relevance (W1; Q1, first part)

We selected these five categories to turn spatial ambiguities and dynamic failure modes reported in prior pointing research into controlled, comparable diagnostic conditions.

Depth overlap and ray ambiguity. Jirak et al. studied pointing confusion caused by object proximity and partial image overlap through real human-robot interaction experiments. CAPE also illustrates interference from other objects along the pointing direction and identifies missing depth information as a cause of selecting the wrong object along that direction. These studies motivate depth overlap and ray ambiguity, which examine target disambiguation under partially overlapping projections and similar pointing directions, respectively. See Jirak et al., 2021 and CAPE, WACV 2026, Fig. 1 and the limitations discussion.

Saliency distraction, temporal confusion, and viewpoint drift. HINT's failure analysis on real-world first-person videos explicitly discusses saliency bias, temporal confusion, and rapid viewpoint drift: models may select a more salient non-target object, confuse references at different times, or struggle to maintain target correspondence after camera motion. These observations directly motivate our saliency-distraction, temporal-confusion, and viewpoint-drift categories. See HINT, CVPR 2026, Appendix B.5.

Building on these identified challenges, we organize the five categories into three forms of spatial competition and two forms of dynamic change. Within a unified target-selection task, we control target-confuser geometry, the gesture sequence, and camera motion to diagnose models category by category rather than reporting overall accuracy alone. Our additional real-world evaluation provides videos corresponding to the five categories, complemented by everyday pointing videos beyond the category-driven setting.

See revised Sec. 3.2 and Appendix A.2 for the taxonomy and its motivation.

2. Effects of motion blur and partial hand occlusion (W1; Q1, second part)

PointMod still achieves higher accuracy than RGB SFT under 5- and 9-pixel motion blur and 2-pixel mosaic perturbations. We apply motion blur or local hand perturbations to the same 500 test videos with 1,500 questions, re-extract WiLoR features from the degraded inputs, and do not retrain the models. The three-model results are:

Input conditionBase InternVL3.5-8BRGB SFTPointMod
Clean input24.93%80.20%88.33%
5-pixel motion blur23.47%76.93%83.67%
9-pixel motion blur21.93%67.33%69.27%
10% hand-box gray patch25.40%79.87%75.60%
2-pixel mosaic25.87%80.40%87.60%
4-pixel mosaic24.33%78.47%66.27%

Under the gray-patch and 4-pixel mosaic conditions, PointMod performs below RGB SFT, indicating some sensitivity to partial degradation of hand information. The 10% gray patch covers the central 10% of the hand-box area; mosaic processing is applied to the central 50% of the hand-box area.

These results also suggest a practical direction: future work could combine PointMod and RGB SFT, using spatial guidance when gesture information is reliable and an RGB SFT branch when it is insufficient, to exploit their complementary behavior under different input conditions.

See revised Appendix C.11, Table C14, and Fig. S8.

AuthorsOpenReview ↗

Response to Reviewer iUBm (2/2)

3. Full videos for the five categories and clarification of the temporal examples (W2, Q2)

The full videos are available at this anonymous demonstration webpage. It includes ten synthetic videos per category and ten real-world videos, each accompanied by the original questions, options, and evaluated answers from the three models, making the complete pointing sequence easier to inspect.

In the temporal-confusion examples, the intended target remains fixed throughout the video. Transient directional alignment during hand movement is a distracting cue, not a change of target. In Fig. 2, solid teal boxes mark targets and dashed orange-red boxes mark confusers. The reference target in Fig. S3(d) is a pot (option A). Identifying the referred object therefore requires considering the full pointing sequence, particularly stable pointing during the hold phase, rather than only local alignment in a transitional frame.

See revised Figs. 2 and S3 for the temporal examples.

4. Pointing-hold frame versus full video (Q3)

The full video outperforms the hold frame alone, showing that the current model can use visual information beyond the hold moment. With model weights fixed, we compare the two inputs on the same 500 videos and 1,500 questions:

InputRGB SFTPointMod
Pointing-hold frame only76.13%78.27%
Full video, 16 frames80.20%88.33%

The full video improves PointMod by 10.07 percentage points and RGB SFT by 4.07 points. In the single-frame condition, the same hold frame is repeated across all 16 input slots, and PointMod's heatmap is generated from that frame alone, ensuring that this condition uses only information from the hold moment.

See revised Appendix C.8 and Table C10.

5. Overlaying the same heatmap on frames versus internal modulation (Q4)

Using the same predicted heatmap, PointMod's internal modulation exceeds our frame-overlay control by 4.73 percentage points. We fix the predicted heatmap from the same Locator, the original 16 frames, and the training QA data. We overlay the heatmap directly on the RGB frames, disable internal modulation, and evaluate after the same 12k downstream updates.

On the full test set of 500 videos and 1,500 questions, the overlay control achieves 83.60% and PointMod achieves 88.33%. This matched control directly compares two ways of incorporating the same spatial guidance and supports the effectiveness of internal visual-feature modulation.

See revised Sec. 5.4 and Table 6; implementation details are in Appendix B.3.

6. Real-world first-person pointing evaluation (W1, Q5)

We newly collected and constructed a real-world first-person pointing test set, on which PointMod improves over RGB SFT by 12.71 percentage points. The test set contains 160 manually annotated videos, with three questions per video and 480 questions in total. Of these, 60 correspond to the five ambiguity categories, with 12 per category. Another 100 were collected around everyday pointing tasks, extending real-world evaluation beyond the predefined categories.

PointMod achieves an overall accuracy of 68.13%, compared with 55.42% for RGB SFT and 22.08% for Base InternVL3.5-8B. The improvements on the five-category diagnostic and everyday-pointing portions are 13.33 and 12.33 percentage points, respectively, providing direct experimental support for PointMod's effectiveness on real-world pointing tasks.

See revised Sec. 5.5 and Table 7; further details and examples are in Appendices A.5 and C.10, Table C13, and Figs. S6-S7.

Thank you again for your thoughtful feedback. We have added experiments and revised the manuscript to address your comments, as detailed in our responses above.

If our responses and additional results address your main concerns, we would be grateful if you could consider updating your overall assessment accordingly. We would also be happy to clarify any remaining questions.

Reviewer_tXMNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PointConfusion, a synthetic egocentric video benchmark where the model must select the object the user is pointing at. Samples are grouped into five ambiguity types. The paper also proposes PointMod: a small Locator converts frozen WiLoR hand features into a single 16×16 heatmap per video, which re-weights the visual tokens after the projector. On InternVL3.5-8B, PointMod reaches 88.33%, compared with 80.20% for RGB fine-tuning under the same LoRA budget.

Critical strengths
  • The per-category design is the most valuable part of the benchmark. Because target and distractor geometry are controlled, the results show where models fail, not just how often.
  • The method is simple and adds no extra tokens. The removal/shuffle test (Table 6) shows the model really depends on the heatmap.
  • The text-only controls (Table C3) are a useful sanity check. That said, the 19.8% name-prior baseline means name shortcuts are not fully ruled out.
Critical weaknesses
  1. All pointing-specific evaluations use synthetic data. The Video-MME subset provides limited evidence about general video QA, but does not directly assess real-world pointing. Transfer to real-world pointing therefore remains unvalidated. Also, 5,000 videos come from 3,382 layouts, so layouts are reused, but the paper does not say whether train and test are separated by layout or object asset.
  2. The Locator is trained on ground-truth target boxes, while RGB SFT, HINT, and CAPE do not use box labels during fine-tuning. Without a baseline that also gets box supervision (e.g., RGB SFT with an auxiliary box loss), it is hard to tell how much of the 8-point gain comes from the modulation design and how much from the extra labels.
  3. In viewpoint drift the camera moves, so the target's image position changes, but the same heatmap is applied to every frame (Fig. S5). Still, viewpoint drift has one of the largest gains (+9.67). I would like to understand why. Is the heatmap mainly matching the hold frame, and how well does it overlap the target in the other frames?
Questions for authors
  1. Are train/test separated by layout? Can you report results on unseen layouts and on real pointing data (e.g., EgoPointVQA or EgoPoint-Bench)?
  2. What happens if RGB SFT gets an auxiliary target-box loss? And can you check the HINT implementation, given the removal result?
  3. How is an eligible confuser defined, which categories do the 12 excluded videos belong to, and can you report Table 5 on the test set?
3 comments
AuthorsOpenReview ↗

Response to Reviewer tXMN (1/2)

Thank you for recognizing the category-wise diagnostic design and the simplicity of PointMod, and for your detailed suggestions. We have clarified the data splits and added supervision comparisons, analyses of the heatmap mechanism, and real-world evaluations, incorporating the results and explanations into the revised manuscript.

1. Layout separation and real-world pointing evaluation (W1, Q1)

Training and test sets are separated by layout, so the existing test set already evaluates unseen layouts; we have also added a direct real-world pointing evaluation. The training, validation, and test sets contain 3,076, 176, and 130 distinct layouts, respectively, with no layout overlap across the three splits. Layout reuse occurs within each split. The separation is by layout rather than by object asset, and object assets are reused across splits.

To add real-world validation, we newly collected and constructed a first-person pointing test set containing 160 manually annotated videos, with three questions per video and 480 questions in total. Of these, 60 videos cover the five ambiguity categories, while another 100 were collected around everyday pointing tasks, extending the evaluation beyond the predefined ambiguity scenarios. PointMod achieves 68.13%, compared with 55.42% for RGB SFT and 22.08% for Base InternVL3.5-8B, improving over RGB SFT by 12.71 percentage points. In addition, on all 834 native four-option questions in EgoPoint-Bench's public real-image task, PointMod achieves 50.6%, compared with 50.0% for RGB SFT and 48.08% for Base InternVL3.5-8B, improving by 0.6 and 2.52 percentage points, respectively.

See revised Sec. 3.3 and Appendix A.1 for the split, and Sec. 5.5 and Table 7 for real-world evaluation.

2. Additional target-box supervision (W2; Q2, first part)

With the same training QA data and 12k downstream updates, RGB SFT with auxiliary box supervision reaches 85.40%, while PointMod achieves 88.33%, a further 2.93-point gain. Following your suggestion, we add an auxiliary target-box loss that updates the LoRA parameters shared with the QA task. The box-prediction branch is removed at inference, and no ground-truth target boxes are provided. All methods use the same training QA data, a budget of 12k downstream updates, and the full test set of 500 videos and 1,500 questions:

MethodAccuracy
RGB SFT80.20%
RGB SFT + auxiliary target-box loss85.40%
Same predicted heatmap overlaid on RGB frames, with internal modulation disabled and downstream fine-tuning83.60%
PointMod88.33%

Auxiliary box supervision improves RGB SFT by 5.20 percentage points, while PointMod remains 2.93 points higher. Furthermore, the frame-overlay control using the same predicted heatmap is 4.73 points below PointMod, providing a more direct interface comparison supporting internal visual-feature modulation.

See revised Sec. 5.4 and Table 6; Appendix B.3 details both controls.

3. Camera motion and the shared heatmap (W3)

The shared heatmap provides coarse spatial guidance rather than requiring precise target tracking in every frame. PointMod aggregates hand-gesture information from 16 time points into video-level spatial guidance, providing an approximate target location, and combines it with the full video to identify the referred object. After camera motion, this cue can still aid recognition as long as it covers some useful target regions.

Across 100 viewpoint-drift videos, the heatmap peak falls inside the target box in 44%, 60%, and 56% of videos at the start of the hold phase, the hold midpoint, and the midpoint of the lowering phase, respectively. Target coverage is therefore not confined to the hold midpoint. PointMod also achieves 90.33% with the full video versus 79.67% with only the hold frame. These results support the interpretation that the shared heatmap supplies a coarse spatial prior, while the full video supplies complementary visual information, jointly helping to address viewpoint drift.

See revised Appendix C.8, Tables C10-C11, and Fig. S5.

AuthorsOpenReview ↗

Response to Reviewer tXMN (2/2)

4. HINT implementation and gesture-removal results (Q2, second part)

We checked the core implementation against the HINT paper, and the hand-feature encoding and frame-wise token insertion follow its description. Specifically, we use WiLoR's 21 3D hand keypoints, encode them through LayerNorm and a two-layer MLP, and insert a gesture token alongside the corresponding frame's visual tokens using a hand-detection confidence threshold of 0.5.

HINT* in our paper is a task-adapted version trained on PointConfusion. Its accuracies with normal inputs and after removing gesture tokens are 77.00% and 77.33%, respectively, indicating that this adapted model has limited dependence on additional gesture tokens for this task. The original authors' public repository has not yet provided the dataset, complete training/evaluation code, or checkpoint. We therefore implemented the core mechanism according to the paper and performed task-adaptation training on PointConfusion; the reported results correspond to this adaptation setting.

See revised Appendix B.4 and Tables B2 and 4.

5. Confuser definition, excluded samples, and test-set Table 5 (Q3)

Definition and eligibility.

A confuser here is the primary non-target object designated during dataset construction to compete ambiguously with the target, selected according to the construction conditions of each category. For example, one depth-overlap sample uses a bowl as the target and a house plant as the primary confuser. They are at different depths but have partially overlapping image projections, creating competition in target identification.

Eligibility further specifies whether this predesignated confuser can participate in the single-frame spatial evaluation: its projected box at the hold midpoint must intersect the image with positive area. The discrimination metric in Table 5 compares the summed heatmap responses inside the target and confuser boxes and reports the proportion for which the target response is greater. Both heatmap methods use the same eligible set.

Excluded samples. In the 12 excluded validation samples, the primary confuser is outside the image at this evaluation frame, preventing the single-frame paired comparison. These samples are therefore excluded only from this metric. Two belong to saliency distraction and ten to viewpoint drift. Objects need not remain in view throughout a video, and all these videos and their QA items remain in the question-answering evaluation.

Full-test results.

HeatmapPeak hit in target boxTarget response greater than confuserMean heatmap mass in target box
Fixed W2F8.60%67.76%7.17%
Learned heatmap H77.40%75.71%28.81%

The paired metric uses 488 validation videos and 490 test videos; the ten excluded test videos all belong to viewpoint drift. Peak hit and in-box heatmap mass use all 500 videos in each split. The learned heatmap outperforms fixed W2F on all three test-set metrics.

See revised Table 5 and Appendix B.5; validation results are in Appendix C.9, Table C12.

6. Name priors and visual dependence (S3)

We further investigated name-prior and answer-distribution shortcuts through visual interventions, showing that RGB SFT's high accuracy depends on visual input matched to the question. On 100 fixed videos with 300 questions, accuracy is 82.67% with the original videos and drops to 8.67% when all 16 frames are replaced with gray images. Keeping the original questions and options but replacing each video with another video from the same category reduces accuracy against the original reference answers to 18.33%. When only the options are permuted and the answer labels are updated accordingly, accuracy remains 82.33%.

Beyond the original 19.8% name-prior baseline, these experiments provide direct evidence of visual dependence and further counter the explanation that high performance mainly comes from object names or fixed answer letters: the model requires visual content matched to the question rather than relying solely on name priors or the answer distribution.

See revised Appendix C.3 and Tables C3-C4.

Thank you again for your thoughtful feedback. We have added experiments and revised the manuscript to address your comments, as detailed in our responses above.

If our responses and additional results address your main concerns, we would be grateful if you could consider updating your overall assessment accordingly. We would also be happy to clarify any remaining questions.

Reviewer_19zeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PointConfusion, an egocentric video understanding benchmark organized around spatial pointing ambiguities. It also proposes PointMod, which maps hand information into spatial heatmaps and provides explicit spatial guidance through visual feature modulation. Experimental results demonstrate that the proposed method outperforms existing MLLMs on this task and improves the performance of task-fine-tuned models across different architectures and model sizes.

Critical strengths
  1. This paper not only introduces PointConfusion, which systematically evaluates five categories of spatial and dynamic pointing ambiguities, but also proposes PointMod, a method that learns spatial reference heatmaps from hand geometry and uses them to guide visual features without introducing additional gesture tokens. This approach achieves better performance than direct RGB fine-tuning.

  2. The main experiments and ablation studies are comprehensive and well presented. The results demonstrate the advantages of PointMod across different categories in the ambiguity taxonomy.

Critical weaknesses
  1. Since PointMod involves LoRA training, could the authors investigate a training-free approach that directly incorporates the hand keypoints extracted by WiLoR and the corresponding heatmap weights into visual tokens? It would be useful to compare the inference performance of this approach with RGB fine-tuning to better distinguish whether the performance gains mainly come from fine-tuning or from PointMod's visual modulation.

  2. Since EgoPointVQA[1] also addresses gesture-based reference understanding, could the authors evaluate whether the fine-tuned PointMod model generalizes to the target reference task in EgoPointVQA?

#1.Choi, Yura, et al. "Do you see what i am pointing at? gesture-based egocentric video question answering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors

Could the authors provide additional generalization experiments on target reference tasks using real-world egocentric benchmarks? Since the current dataset consists of synthetic videos, differences between simulated and real-world visual environments may affect pointing-based target reference understanding.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 19ze

Thank you for carefully reviewing our work and for your constructive suggestions. We have added heatmap-injection comparisons with frozen VLMs and evaluations on real-world videos and public external data, and incorporated the results and implementation details into the revised manuscript.

1. Is directly incorporating a heatmap effective without additional VLM fine-tuning? (W1)

Yes. Direct heatmap injection improves accuracy without additional VLM fine-tuning, while PointMod achieves higher accuracy by combining spatial modulation with downstream adaptation. We fix the model weights and vary only whether the heatmap is incorporated, using the same 500 test videos and 1,500 questions. All conditions use the same trained Locator; thus, "without additional fine-tuning" refers to the VLM.

VLM weightsWithout heatmapWith heatmap
Base InternVL3.5-8B (without loading LoRA)24.93%36.00%
RGB SFT80.20%80.47%
PointMod33.47%88.33%

Incorporating the heatmap improves Base InternVL3.5-8B by 11.07 percentage points, whereas temporarily incorporating it into RGB SFT yields only a 0.27-point improvement. Full PointMod improves over RGB SFT by 8.13 percentage points, supporting the design that combines spatial modulation with downstream adaptation. The PointMod result without a heatmap is an inference-time removal diagnostic of the trained model.

See revised Appendix C.7 and Table C8.

2. Generalization evaluation on EgoPointVQA (W2)

We have not yet obtained the complete data and supporting resources required for this evaluation through EgoPointVQA's official public channels. The official repository still lists the dataset, checkpoint, and training and evaluation code as pending releases. We therefore do not report results on this dataset in this response.

As a complementary evaluation on public external data, we evaluate all 834 native four-option questions in EgoPoint-Bench's real-image task. PointMod achieves 50.6%, compared with 50.0% for RGB SFT and 48.08% for Base InternVL3.5-8B, improving by 0.6 and 2.52 percentage points, respectively.

The complementary EgoPoint-Bench evaluation is reported in revised Sec. 5.5 and Table 7.

3. How does PointMod perform on real-world first-person pointing videos? (Q1)

We newly collected and constructed a real-world first-person pointing test set, on which PointMod improves over RGB SFT by 12.71 percentage points. The test set contains 160 manually annotated videos, with three questions per video and 480 questions in total. Of these, 60 videos cover the five ambiguity categories, with 12 per category, to diagnose these challenges in real scenes. Another 100 videos were collected around everyday pointing tasks, extending the evaluation to practical pointing situations beyond the predefined categories.

PointMod achieves an overall accuracy of 68.13%, compared with 55.42% for RGB SFT and 22.08% for Base InternVL3.5-8B. The improvements on the five-category diagnostic and everyday-pointing portions are 13.33 and 12.33 percentage points, respectively. These results provide direct experimental support for PointMod's effectiveness on real-world pointing videos.

See revised Sec. 5.5 and Table 7; Appendix A.5 describes Real160, and Appendix C.10 provides Table C13 and Figs. S6-S7.

Thank you again for your thoughtful feedback. We have added experiments and revised the manuscript to address your comments, as detailed in our responses above.

If our responses and additional results address your main concerns, we would be grateful if you could consider updating your overall assessment accordingly. We would also be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.