- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes EviGaze, which investigates whether multimodal models judge gaze function based on interactive evidence by separately evaluating gaze target recognition, evidence localization, and functional judgment, and by removing annotated subsequent interactive evidence. Experiments show that models exhibit weaker sensitivity to evidence removal compared to humans. The authors further explain this phenomenon as insufficient evidence utilization through prompt repair, hidden-state probes, and evidence-focused prompting.
Critical strengths
- The research question is clear and valuable. The paper distinguishes between "knowing where a person is looking" and "understanding the role of that glance in an interaction," and further examines whether correct answers are grounded in relevant evidence.
- The experiments are comprehensive. Beyond reporting accuracy under full inputs, they provide complementary diagnostics such as evidence removal, neutral removal, abstention, and natural pairing to examine model reliance on interactive evidence from multiple perspectives.
- The prompting experiments include structurally similar placebos and compare the differential impacts of evidence removal versus irrelevant removal on answer stability, which helps distinguish improvements driven by evidence focusing from the effects of general step-by-step prompting.
Critical weaknesses
-
The paper regards maintaining original labels after evidence removal as EGF (evidence grounding failure). Thus, there is an important premise: after deleting the annotated segments, there should no longer be sufficient information in the video to support the original judgment. However, Appendix A.5 ends the evidence interval at the moment "the function first becomes determinable." While this marks the earliest point where the function can be confirmed, it does not necessarily cover all subsequent content that could support the same judgment. If subsequent actions continue or follow-up dialogue reconfirms what happened, the model might still make the same judgment based on this content. The authors' human abstention experiments demonstrate that the deletion operation indeed makes many videos difficult to judge. Nevertheless, I wonder how many of the events where models retained their original labels fall into this scenario. Have the authors checked the remaining content of the trimmed videos? Reporting EGF specifically on events confirmed to lack sufficient judgmental basis would more clearly clarify the extent to which the model retaining its answers reflects actual grounding failure.
-
The authors find that even when provided with the correct gaze target and evidence time interval, more than half of the original errors cannot be fixed, supporting the explanation that "evidence utilization is the primary bottleneck." However, telling the model who to look at and which segment to watch does not guarantee it understands what happened in that segment. If the model misrecognizes a person's action or response, it will still make a wrong judgment even if it finds the correct location. Hidden-state probes could have helped distinguish between these two scenarios, but current results are reported as aggregate summaries across all events. I would like to know whether the probes can still decode the correct function in events where the model actually answers incorrectly, especially those that joint prompting still fails to fix. If the probes only succeed on events the model would have answered correctly anyway, they fail to explain these remaining errors. The issue here is not demanding that probes prove internal causal routing, but rather confirming whether decodable information truly exists in the failure cases that need to be explained. The evidence-focused prompting already requires models to describe segment contents; the authors could first check whether these descriptions are accurate, and then analyze them alongside the probe results for corresponding events. If the model can correctly describe the interaction and the probes can decode the correct function, yet it still makes the wrong final choice, this would more strongly support the explanation that "the evidence was understood but did not sufficiently influence the judgment."
Questions for authors
As described above.
3 comments
Responses to Reviewer p227 (Part II)
W2: The oracle-hint design localizes the stage at which errors arise (also noted by Reviewer 27yg). We agree that perception of the located evidence calls for a direct test, which we now provide. Following your suggestions, we have conducted the requested analyses.
(1) Probe decoding on failure cases. We group the held-out predictions of the local evidence probe by the outcome of the same model's baseline answer on the same event. The results are reported below, in Table 3(d), the Resid. column of Table 3(a), and Table 32 of the revised paper, and in Section 5.2.
| Model | L3 correct | L3 incorrect | Combined-hint residual | Perm. (residual) |
|---|---|---|---|---|
| Qwen3-VL-2B | 63.0 | 52.2 | 50.9 | 17.8 |
| Qwen3-VL-8B | 67.6 | 58.2 | 55.4 | 17.1 |
| Qwen3-VL-8B-Thinking | 72.8 | 59.8 | 58.2 | 16.7 |
| InternVL3.5-2B | 61.5 | 52.2 | 51.5 | 17.9 |
| InternVL3.5-8B | 69.3 | 61.0 | 56.8 | 16.2 |
| MiniCPM-V-4.6 | 67.8 | 52.6 | 51.4 | 18.1 |
| LLaVA-Video-7B | 73.5 | 65.1 | 64.5 | 16.4 |
| Macro mean | 67.9 | 57.3 | 55.5 | 17.2 |
On the 2,676 errors that combined target and interval hints leave unrepaired, the probe recovers the correct function in 55.5% of cases, more than three times the permuted-label control, and above 50% in all seven models. The deciding information therefore remains present in most of the errors that need to be explained.
(2) Accuracy of the evidence descriptions. From the 1,243 model-event cells in which the evidence-gated prompt localizes the evidence but returns a wrong function (Appendix I), we sample 300 with stratification by model. Two raters view the annotated interval and independently rate each OBSERVATION field as matching, partially matching, or not matching the observable interaction. A senior adjudicator resolves disagreements (raw agreement 86.7%, Cohen's κ = 0.73). The descriptions match the interaction in 66.3% of cases, partially match in 23.0%, and do not match in 10.7% (Table 35, Section 5.3).
(3) Joint analysis. Crossing the description rating with the probe outcome on the same 300 cells gives the following distribution (Table 3(c) and Table 36).
| Probe correct | Probe incorrect | |
|---|---|---|
| Description matches | 38.0% (114) | 28.3% (85) |
| Partial or mismatch | 18.7% (56) | 15.0% (45) |
In 38.0% of these errors, the model describes the interaction correctly and its internal state encodes the correct function, yet the final choice is still wrong. This is the case you identified as the strongest support for the evidence-use account, and it forms the largest cell in all seven models. Errors in which both the description and the probe fail, the clearest sign of misperception, account for only 15.0%. Together with the oracle-hint results, these analyses show that most residual errors follow a correct perception of the evidence.
Gentle Reminder on Our Responses
Dear Reviewer p227,
Thank you again for your detailed and constructive review. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.
Following your suggestions, we evaluated EGF on the 353 events that humans confirm as undecidable after removal, where the models still retain their labels on 71.5% of eligible events. We also showed that the local probe decodes the correct function on 55.5% of the errors left unrepaired by the hints, and that 38.0% of the correctly localized errors combine a matching description with a correct probe decoding. These results are included in the revised paper (Tables 2(d), 3(c), and 3(d), and Appendix O).
We would be happy to clarify any remaining questions.
Best regards, The Authors
Responses to Reviewer p227 (Part I)
To Reviewer p227: Thanks for the constructive comments and for recognizing the value of the research question and the comprehensiveness of the experiments. Responses to specific comments are given below, which will also be added to the revised paper.
W1: We agree that EGF rests on the premise that the edited clip no longer supports the original judgment. We have verified this premise at three levels.
(1) First, at construction. During manual screening, annotators inspected the content after each evidence interval. Events in which later actions or dialogue reconfirm the function were excluded because their evidence cannot be cleanly removed (Table 5). In the revised paper, Section 3.3 and Appendix A.2 state this criterion explicitly, and Section 3.4 states the premise itself.
(2) Second, at the aggregate level. After removal, human accuracy falls from 94.4% to 30.0% (Table 1), and humans choose cannot-be-determined (CD) in 74.6% of responses, against 3.9% after neutral deletion (Table 2a).
(3) Finally and most importantly, at the event level, as you suggested. We select the 353 events on which both human abstention responses to the edited version choose CD, so that the trimmed video is confirmed to lack sufficient basis for a judgment. On these events, human accuracy after removal falls to 16.8%, the chance level. We then recompute the EGF rate of all fifteen models on this subset. The results are reported below, in Table 2(d) and Table 30 of the revised paper, and in Section 4.3.
On events confirmed to lack a judgmental basis, the models still retain their labels on 71.5% of eligible events. Every model stays within 4.5 points of its rate on all paired events. The retained labels therefore reflect grounding failure, not later content that supports the same judgment. Your suggestion turns the premise of EGF into a directly measured property of the benchmark.