- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces PointConfusion, an egocentric video benchmark for controlled pointing gesture grounding in multiple choice question answering format. The dataset controls 5 pointing ambiguity categories: depth overlap, ray ambiguity, saliency distraction, temporal confusion, and viewpoint drift, 15,000 total questions from 5,000 synthetic videos, where 1,500 questions are used for test.
The authors also propose PointMod method, which predicts a heatmap from reconstructed 107-dimension hand information, supervised by target bounding boxes. This heatmap is used to modulate visual tokens before putting to LLM. On PointConfusion, an experiment with InternVL3.5-8B showed that PointMod improves 8.3% compared with standard finetuning.
Critical strengths
-
PointConfusion introduces a structured types of pointing ambiguities, and constructs videos with controlled conditions. This category-wise evaluation helps identifying which types of ambituity is challenging for existing MLLMs.
-
PointMod provides a simple but effective way to inject pointing geature information into visual representations. It's relative gain compared with RGB finetuning suggests that explicit spatial guidance is useful.
Critical weaknesses
-
The connection to real-world pointing gesture understandig is unclear. Five conditions are motivated by prior work, but the paper does not show how representative they are of the challenges in real world pointing videos. For example, motion blur or partial hand occlusion can make hand reconstruction unreliable, potentially affecting PointMod's spatial guidance. While the controlled synthetic setup is useful for isolating specific ambiguities, PointMod's performance on real-world pointing gestures remains unclear. A small real-world evaluation or further discussion of these limitations would strengthen the practical relevance of the work.
-
Generated gesture videos are hard to assess from frame visualizations. For example, in Figure 2 and S3, Temporal confusion example made a bit difficult to follow which of the two objects are referred.
Questions for authors
- Could the authors explain on the selection of the five ambiguity categories and their relevance to real-world pointing scenarios? In particular, have the authors considered the effects of motion blur or hand occlusion on PointMod, given its reliance on reconstructed hand features?
- Could the authors provide video clips illustrating the five ambiguity types?
- Could the authors compare performance using on a pointing-hold frame against the full video? it might help
- Could the authors compare PointMod with another way of using the same learned heatmap, such as overlaying it on the video frames? This would help separating the benefits from heatmap from visual modulation.
- Is it possible to evaluate PointMod on a small set of real-world pointing videos?
5 comments
Response to Reviewer iUBm (2/2)
3. Full videos for the five categories and clarification of the temporal examples (W2, Q2)
The full videos are available at this anonymous demonstration webpage. It includes ten synthetic videos per category and ten real-world videos, each accompanied by the original questions, options, and evaluated answers from the three models, making the complete pointing sequence easier to inspect.
In the temporal-confusion examples, the intended target remains fixed throughout the video. Transient directional alignment during hand movement is a distracting cue, not a change of target. In Fig. 2, solid teal boxes mark targets and dashed orange-red boxes mark confusers. The reference target in Fig. S3(d) is a pot (option A). Identifying the referred object therefore requires considering the full pointing sequence, particularly stable pointing during the hold phase, rather than only local alignment in a transitional frame.
See revised Figs. 2 and S3 for the temporal examples.
4. Pointing-hold frame versus full video (Q3)
The full video outperforms the hold frame alone, showing that the current model can use visual information beyond the hold moment. With model weights fixed, we compare the two inputs on the same 500 videos and 1,500 questions:
| Input | RGB SFT | PointMod |
|---|---|---|
| Pointing-hold frame only | 76.13% | 78.27% |
| Full video, 16 frames | 80.20% | 88.33% |
The full video improves PointMod by 10.07 percentage points and RGB SFT by 4.07 points. In the single-frame condition, the same hold frame is repeated across all 16 input slots, and PointMod's heatmap is generated from that frame alone, ensuring that this condition uses only information from the hold moment.
See revised Appendix C.8 and Table C10.
5. Overlaying the same heatmap on frames versus internal modulation (Q4)
Using the same predicted heatmap, PointMod's internal modulation exceeds our frame-overlay control by 4.73 percentage points. We fix the predicted heatmap from the same Locator, the original 16 frames, and the training QA data. We overlay the heatmap directly on the RGB frames, disable internal modulation, and evaluate after the same 12k downstream updates.
On the full test set of 500 videos and 1,500 questions, the overlay control achieves 83.60% and PointMod achieves 88.33%. This matched control directly compares two ways of incorporating the same spatial guidance and supports the effectiveness of internal visual-feature modulation.
See revised Sec. 5.4 and Table 6; implementation details are in Appendix B.3.
6. Real-world first-person pointing evaluation (W1, Q5)
We newly collected and constructed a real-world first-person pointing test set, on which PointMod improves over RGB SFT by 12.71 percentage points. The test set contains 160 manually annotated videos, with three questions per video and 480 questions in total. Of these, 60 correspond to the five ambiguity categories, with 12 per category. Another 100 were collected around everyday pointing tasks, extending real-world evaluation beyond the predefined categories.
PointMod achieves an overall accuracy of 68.13%, compared with 55.42% for RGB SFT and 22.08% for Base InternVL3.5-8B. The improvements on the five-category diagnostic and everyday-pointing portions are 13.33 and 12.33 percentage points, respectively, providing direct experimental support for PointMod's effectiveness on real-world pointing tasks.
See revised Sec. 5.5 and Table 7; further details and examples are in Appendices A.5 and C.10, Table C13, and Figs. S6-S7.
Thank you again for your thoughtful feedback. We have added experiments and revised the manuscript to address your comments, as detailed in our responses above.
If our responses and additional results address your main concerns, we would be grateful if you could consider updating your overall assessment accordingly. We would also be happy to clarify any remaining questions.
Response to Reviewer iUBm (1/2)
Thank you for recognizing the diagnostic value of PointConfusion and the spatial-guidance design of PointMod. Following your suggestions, we have strengthened the literature grounding for the five ambiguity categories, added robustness evaluations and video demonstrations, and incorporated the experiments and explanations into the revised manuscript.
1. Selection of the five ambiguity categories and their real-world relevance (W1; Q1, first part)
We selected these five categories to turn spatial ambiguities and dynamic failure modes reported in prior pointing research into controlled, comparable diagnostic conditions.
Depth overlap and ray ambiguity. Jirak et al. studied pointing confusion caused by object proximity and partial image overlap through real human-robot interaction experiments. CAPE also illustrates interference from other objects along the pointing direction and identifies missing depth information as a cause of selecting the wrong object along that direction. These studies motivate depth overlap and ray ambiguity, which examine target disambiguation under partially overlapping projections and similar pointing directions, respectively. See Jirak et al., 2021 and CAPE, WACV 2026, Fig. 1 and the limitations discussion.
Saliency distraction, temporal confusion, and viewpoint drift. HINT's failure analysis on real-world first-person videos explicitly discusses saliency bias, temporal confusion, and rapid viewpoint drift: models may select a more salient non-target object, confuse references at different times, or struggle to maintain target correspondence after camera motion. These observations directly motivate our saliency-distraction, temporal-confusion, and viewpoint-drift categories. See HINT, CVPR 2026, Appendix B.5.
Building on these identified challenges, we organize the five categories into three forms of spatial competition and two forms of dynamic change. Within a unified target-selection task, we control target-confuser geometry, the gesture sequence, and camera motion to diagnose models category by category rather than reporting overall accuracy alone. Our additional real-world evaluation provides videos corresponding to the five categories, complemented by everyday pointing videos beyond the category-driven setting.
See revised Sec. 3.2 and Appendix A.2 for the taxonomy and its motivation.
2. Effects of motion blur and partial hand occlusion (W1; Q1, second part)
PointMod still achieves higher accuracy than RGB SFT under 5- and 9-pixel motion blur and 2-pixel mosaic perturbations. We apply motion blur or local hand perturbations to the same 500 test videos with 1,500 questions, re-extract WiLoR features from the degraded inputs, and do not retrain the models. The three-model results are:
Under the gray-patch and 4-pixel mosaic conditions, PointMod performs below RGB SFT, indicating some sensitivity to partial degradation of hand information. The 10% gray patch covers the central 10% of the hand-box area; mosaic processing is applied to the central 50% of the hand-box area.
These results also suggest a practical direction: future work could combine PointMod and RGB SFT, using spatial guidance when gesture information is reliable and an RGB SFT branch when it is insufficient, to exploit their complementary behavior under different input conditions.
See revised Appendix C.11, Table C14, and Fig. S8.