acceptodds
Under review as a conference paper at ICLR 2027

Knowing Where Is Not Knowing What: Evidence-Grounding Failure in Social Gaze

Abstract

Understanding social gaze is essential for socially interactive agents, yet a look's function depends less on its direction than on the subsequent interaction. While multimodal large language models (MLLMs) are evaluated on gaze following, whether they ground functional judgments in this interaction remains largely unexplored. To address this gap, we present **EviGaze**, a benchmark specifically designed to diagnose evidence grounding in social gaze understanding. EviGaze annotates 5,120 gaze events from four social-video corpora with their *function* and *evidence interval*. Since this interval follows the gaze, removing it withdraws the grounds for judgment while keeping the look visible. We define *evidence-grounding failure* (EGF) as label retention after removal once the target and evidence are located. Extensive experiments are conducted to benchmark fifteen MLLMs on EviGaze. Experimental results reveal that EGF persists across all fifteen MLLMs, with evidence removal lowering model accuracy by only 16.0 percentage points against 64.4 for humans. Probing reveals that model states encode functional information that judgments underuse, a dissociation consistent across all seven analyzed models. Prompting models to describe the evidence before answering reduces the EGF rate beyond a matched placebo. The benchmark and code will be publicly released to support further research.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Social Gaze Understanding, Evidence Grounding, Video Understanding, Representation Probing, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p227OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EviGaze, which investigates whether multimodal models judge gaze function based on interactive evidence by separately evaluating gaze target recognition, evidence localization, and functional judgment, and by removing annotated subsequent interactive evidence. Experiments show that models exhibit weaker sensitivity to evidence removal compared to humans. The authors further explain this phenomenon as insufficient evidence utilization through prompt repair, hidden-state probes, and evidence-focused prompting.

Critical strengths
  1. The research question is clear and valuable. The paper distinguishes between "knowing where a person is looking" and "understanding the role of that glance in an interaction," and further examines whether correct answers are grounded in relevant evidence.

  2. The experiments are comprehensive. Beyond reporting accuracy under full inputs, they provide complementary diagnostics such as evidence removal, neutral removal, abstention, and natural pairing to examine model reliance on interactive evidence from multiple perspectives.

  3. The prompting experiments include structurally similar placebos and compare the differential impacts of evidence removal versus irrelevant removal on answer stability, which helps distinguish improvements driven by evidence focusing from the effects of general step-by-step prompting.
Critical weaknesses
  1. The paper regards maintaining original labels after evidence removal as EGF (evidence grounding failure). Thus, there is an important premise: after deleting the annotated segments, there should no longer be sufficient information in the video to support the original judgment. However, Appendix A.5 ends the evidence interval at the moment "the function first becomes determinable." While this marks the earliest point where the function can be confirmed, it does not necessarily cover all subsequent content that could support the same judgment. If subsequent actions continue or follow-up dialogue reconfirms what happened, the model might still make the same judgment based on this content.


The authors' human abstention experiments demonstrate that the deletion operation indeed makes many videos difficult to judge. Nevertheless, I wonder how many of the events where models retained their original labels fall into this scenario. Have the authors checked the remaining content of the trimmed videos? Reporting EGF specifically on events confirmed to lack sufficient judgmental basis would more clearly clarify the extent to which the model retaining its answers reflects actual grounding failure.


  2. The authors find that even when provided with the correct gaze target and evidence time interval, more than half of the original errors cannot be fixed, supporting the explanation that "evidence utilization is the primary bottleneck." However, telling the model who to look at and which segment to watch does not guarantee it understands what happened in that segment. If the model misrecognizes a person's action or response, it will still make a wrong judgment even if it finds the correct location.


Hidden-state probes could have helped distinguish between these two scenarios, but current results are reported as aggregate summaries across all events. I would like to know whether the probes can still decode the correct function in events where the model actually answers incorrectly, especially those that joint prompting still fails to fix. If the probes only succeed on events the model would have answered correctly anyway, they fail to explain these remaining errors. The issue here is not demanding that probes prove internal causal routing, but rather confirming whether decodable information truly exists in the failure cases that need to be explained.


The evidence-focused prompting already requires models to describe segment contents; the authors could first check whether these descriptions are accurate, and then analyze them alongside the probe results for corresponding events. If the model can correctly describe the interaction and the probes can decode the correct function, yet it still makes the wrong final choice, this would more strongly support the explanation that "the evidence was understood but did not sufficiently influence the judgment."

Questions for authors

As described above.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer p227 (Part I)

To Reviewer p227: Thanks for the constructive comments and for recognizing the value of the research question and the comprehensiveness of the experiments. Responses to specific comments are given below, which will also be added to the revised paper.

W1: We agree that EGF rests on the premise that the edited clip no longer supports the original judgment. We have verified this premise at three levels.

(1) First, at construction. During manual screening, annotators inspected the content after each evidence interval. Events in which later actions or dialogue reconfirm the function were excluded because their evidence cannot be cleanly removed (Table 5). In the revised paper, Section 3.3 and Appendix A.2 state this criterion explicitly, and Section 3.4 states the premise itself.

(2) Second, at the aggregate level. After removal, human accuracy falls from 94.4% to 30.0% (Table 1), and humans choose cannot-be-determined (CD) in 74.6% of responses, against 3.9% after neutral deletion (Table 2a).

(3) Finally and most importantly, at the event level, as you suggested. We select the 353 events on which both human abstention responses to the edited version choose CD, so that the trimmed video is confirmed to lack sufficient basis for a judgment. On these events, human accuracy after removal falls to 16.8%, the chance level. We then recompute the EGF rate of all fifteen models on this subset. The results are reported below, in Table 2(d) and Table 30 of the revised paper, and in Section 4.3.

ModelΔ (subset)EGF (subset)Δ (all)EGF (all)
GPT-4o−19.868.1−18.567.9
GPT-5.6 Sol−25.662.6−24.561.6
Gemini 3.1 Pro−18.968.2−16.371.4
Claude Opus 5−20.164.8−21.664.8
Qwen3-VL-2B−14.277.6−10.380.0
Qwen3-VL-8B−17.373.9−13.874.6
Qwen3-VL-8B-Thinking−22.469.8−19.368.3
InternVL3.5-2B−10.776.9−9.281.4
InternVL3.5-4B−11.878.5−12.476.6
InternVL3.5-8B−16.775.6−15.173.1
MiniCPM-V-4.6−16.477.0−12.077.9
GLM-4.6V−17.269.7−16.470.7
LLaVA-Video-7B−14.473.0−12.675.4
VideoLLaMA 3−17.869.9−17.469.1
Video-R1−23.266.3−21.364.2
Macro mean−17.871.5−16.071.8
Human−77.5n/a−64.4n/a

On events confirmed to lack a judgmental basis, the models still retain their labels on 71.5% of eligible events. Every model stays within 4.5 points of its rate on all paired events. The retained labels therefore reflect grounding failure, not later content that supports the same judgment. Your suggestion turns the premise of EGF into a directly measured property of the benchmark.

AuthorsOpenReview ↗

Responses to Reviewer p227 (Part II)

W2: The oracle-hint design localizes the stage at which errors arise (also noted by Reviewer 27yg). We agree that perception of the located evidence calls for a direct test, which we now provide. Following your suggestions, we have conducted the requested analyses.

(1) Probe decoding on failure cases. We group the held-out predictions of the local evidence probe by the outcome of the same model's baseline answer on the same event. The results are reported below, in Table 3(d), the Resid. column of Table 3(a), and Table 32 of the revised paper, and in Section 5.2.

ModelL3 correctL3 incorrectCombined-hint residualPerm. (residual)
Qwen3-VL-2B63.052.250.917.8
Qwen3-VL-8B67.658.255.417.1
Qwen3-VL-8B-Thinking72.859.858.216.7
InternVL3.5-2B61.552.251.517.9
InternVL3.5-8B69.361.056.816.2
MiniCPM-V-4.667.852.651.418.1
LLaVA-Video-7B73.565.164.516.4
Macro mean67.957.355.517.2

On the 2,676 errors that combined target and interval hints leave unrepaired, the probe recovers the correct function in 55.5% of cases, more than three times the permuted-label control, and above 50% in all seven models. The deciding information therefore remains present in most of the errors that need to be explained.

(2) Accuracy of the evidence descriptions. From the 1,243 model-event cells in which the evidence-gated prompt localizes the evidence but returns a wrong function (Appendix I), we sample 300 with stratification by model. Two raters view the annotated interval and independently rate each OBSERVATION field as matching, partially matching, or not matching the observable interaction. A senior adjudicator resolves disagreements (raw agreement 86.7%, Cohen's κ = 0.73). The descriptions match the interaction in 66.3% of cases, partially match in 23.0%, and do not match in 10.7% (Table 35, Section 5.3).

(3) Joint analysis. Crossing the description rating with the probe outcome on the same 300 cells gives the following distribution (Table 3(c) and Table 36).

Probe correctProbe incorrect
Description matches38.0% (114)28.3% (85)
Partial or mismatch18.7% (56)15.0% (45)

In 38.0% of these errors, the model describes the interaction correctly and its internal state encodes the correct function, yet the final choice is still wrong. This is the case you identified as the strongest support for the evidence-use account, and it forms the largest cell in all seven models. Errors in which both the description and the probe fail, the clearest sign of misperception, account for only 15.0%. Together with the oracle-hint results, these analyses show that most residual errors follow a correct perception of the evidence.

AuthorsOpenReview ↗

Gentle Reminder on Our Responses

Dear Reviewer p227,

Thank you again for your detailed and constructive review. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.

Following your suggestions, we evaluated EGF on the 353 events that humans confirm as undecidable after removal, where the models still retain their labels on 71.5% of eligible events. We also showed that the local probe decodes the correct function on 55.5% of the errors left unrepaired by the hints, and that 38.0% of the correctly localized errors combine a matching description with a correct probe decoding. These results are included in the revised paper (Tables 2(d), 3(c), and 3(d), and Appendix O).

We would be happy to clarify any remaining questions.

Best regards, The Authors

Reviewer_27ygOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new social gaze benchmark called EviGaze to probe evidence grounding failure (EGF) prone in existing multimodal LLMs (MLLMs). EviGaze consists of 5,120 gaze events sourced from social video datasets, along with annotations for evidence intervals, controlled interventions, and diagnostic questions. The authors use EviGaze to benchmark fifteen MLLMs, ranging from open source MLLMs to proprietary ones, and find that EGF is prevalent in all benchmarked MLLMs in varying degrees, whereas humans are much more proficient at evidence based grounding. Further analysis in the paper show that internal states of the MLLMs retain the social gaze information which is mostly missing in the final output.

Critical strengths
  1. The paper is well-motivated and addresses a very critical problem in MLLMs - understanding the intricacies of social behavior in humans via gaze.

  2. I found the intervention design very interesting and well-thought out. I particularly want to commend the authors for pairing the interventions with the oracle target and interval hints to localize the root cause of EGF within MLLMs.

  3. The paper is very well-written. The details of the benchmark dataset curation are very well presented with easy to follow diagrams and text. Ample details about the benchmarked MLLMs and dataset collection are provided in the appendix that I truly appreciate.

Critical weaknesses

None

Questions for authors
  1. I am curious if the authors considered the fact that the MLLMs they benchmarked have already ingested the videos in the benchmark during training and are simply regurgitating from memory?
3 comments
AuthorsOpenReview ↗

Responses to Reviewer 27yg

To Reviewer 27yg: We are grateful for your positive assessment and your appreciation of the intervention design and the presentation. Responses to the specific question are given below, which will also be added to the revised paper.

Q1: Since EviGaze builds on public corpora, the models may have seen the raw videos during pretraining. Three points clarify how such exposure bears on our findings.

(1) First, memorization would not invalidate EGF but would constitute one form of it. EGF asks whether a judgment rests on the supplied input. A model that restores a removed exchange from memory still commits to a function without the evidence in front of it, which is precisely the failure EviGaze is designed to reveal.

(2) Second, the behavioural evidence points to dialogue priors as the main source of retained labels. Residual errors coincide with sanitized text-only predictions in 72.4% of cases (Appendix I). Two looks from the same source video with different functions receive the same label in 34.7% of the 246 natural pairs (Table 2b), although recalling the video would help a model tell them apart. The models also process the edited input itself: their CD rate rises from 5.9% after neutral deletion to 24.8% after evidence removal (Table 2a), and the sequence-end probe loses 25.8 points of balanced accuracy (Table 3b).

(3) Third, we report the EGF rate by source corpus below, in Table 33, and in Table 2(e) of the revised paper. The macro EGF rate ranges only from 70.6% to 72.7%, so no single corpus drives the result.

VACATIONVideoAttentionTargetVideoCoAttVSGazeAll
Macro EGF72.370.672.771.071.8

We have added this discussion, including the possibility of pretraining exposure, to Section 4.4 and Appendix O.4 of the revised paper.

I thank the authors for the clarification and additional metrics. I wish them good luck with the submission.

AuthorsOpenReview ↗

Thank You for Your Support

Thank you for your kind words and for your support of our work. The per-corpus results and the discussion of pretraining exposure will be kept in the final version.

Reviewer_H8bpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EviGaze, which is a benchmark to evaluate if MLLMs can ground social gaze interpretation in subsequent interaction. The problem is interesting, novel, and relatively underexplored. The authors construct 5,120 annotated gaze events from four public corpora and evaluate 15 MLLMs. There is substantial annotation effort here, with reasonably strong inter-annotator agreement. Overall, I think this is a meaningful empirical contribution and recommend acceptance.

Critical strengths

Substantial and systematic effort to create the annotation and the benchmark, including 35 human participants and 15 MLLMs. This is a meaningful contribution to the field.

Critical weaknesses

I think the main weakness is with the proposed evidence-grounding failure (EGF) metric. EGF measures label retention after evidence removal, but this is not necessarily equivalent to a grounding failure. A model can, for example, incorporate the removal of evidence, but still give the same prediction label because the remaining context still supports that label. Therefore, the metric may overestimate grounding failures. In Figure 1, for example, the gaze, speech, and response remain visible after the money exchange is removed. These observations may still support the original function label, even if the model's more specific explanation is no longer justified. EGF also does not require the original function judgment to be correct. I would like to see a more direct validation of the metric, perhaps by examining cases where human participants overwhelmingly judge the remaining evidence insufficient to determine the function.

Another weakness is the limited visual input. Models receive only 32 sampled frames, and Appendix C.5 shows that EGF decreases substantially as more evidence frames are available. This makes me wonder whether insufficient evidence perception accounts for some of the observed failures. Humans, on the other hand, watch video clips with optional replays, so the comparison is not entirely controlled. An ablation with denser frame sampling would be useful.

That said, the authors have included several thoughtful controls, especially the neutral-deletion and abstention experiments, which strengthen the main findings. My concerns are largely about the interpretation rather than the empirical observations themselves. The main claims of the paper are meaningful and useful.

Minor point that does not influence the score: it is fine that the paper text is AI-generated, but some language is convoluted and hinders understanding. I recommend screening it again to improve the flow of the paper.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Responses to Reviewer H8bp

To Reviewer H8bp: We appreciate your constructive comments and positive assessment. Responses to specific comments are given below, which will also be added to the revised paper.

W1.1: We agree that label retention would overstate grounding failure if the remaining context still supported the original label. EviGaze addresses this in three ways.

(1) First, by construction. The evidence interval covers the observable signals that decide the function and never overlaps the gaze. During manual screening, events whose function is reconfirmed by later actions or dialogue were excluded because their evidence cannot be cleanly removed (Table 5). Section 3.3 and Appendix A.2 of the revised paper state this criterion explicitly.

(2) Second, by the human reference in the initial version. After removal, human accuracy falls from 94.4% to 30.0% (Table 1), and humans choose cannot-be-determined (CD) in 74.6% of responses, against 3.9% after neutral deletion (Table 2a).

(3) Finally and most importantly, following your suggestion, we evaluate the 353 events on which both human abstention responses to the edited version select CD. The results are reported below, in Table 2(d) and Table 30 of the revised paper, and in Section 4.3.

Event setHuman acc. after removalHuman ΔModel ΔModel EGF
All paired events30.0−64.4−16.071.8
Undecidable for humans (353 events)16.8−77.5−17.871.5

On events that humans cannot judge at all, the models keep their labels on 71.5% of eligible events. The EGF rate of every model stays within 4.5 points of its rate on all paired events. Label retention on EviGaze therefore reflects grounding failure, not residual support for the original label.

W1.2: We would like to clarify that this is a deliberate design. EGF measures whether an answer responds to the removal of its evidence, which applies to correct and incorrect answers alike. Table 1 also reports accuracy retention and Δ, which take correctness into account and lead to the same conclusion. To quantify the effect of this design, we split each model's eligible set by the correctness of its intact answer (Table 31, Section 3.4). The EGF rate is 77.5% on correct intact judgments and 49.9% on incorrect ones, and all fifteen models show a higher rate on correct judgments. Including incorrect judgments thus makes the reported EGF rate conservative.

W1.3: We agree with you on Figure 1. In the original example, P1's response remains visible after removal and could still support the request label. We have replaced Figure 1 with one of the 353 events above and updated Section 4.3 accordingly. The gazer looks at a crouching partner, who answers with a fist bump. After the fist bump is removed, both participants stand still, human participants judge the function undecidable, and all three models retain the original label. Qwen3-VL narrates the removed fist bump back, while GPT-5.6 Sol and Claude Opus 5 justify the label from the conversation and from the look itself.

W2: In the initial version, the EGF rate is computed only on events where the model localizes the evidence interval (L2 tIoU ≥ 0.5). It remains 66.1% on events with at least three sampled evidence frames (Appendix C.5). Following your suggestion, we further rerun all L1, L2, and L3 queries of the seven diagnostic models with 64 frames at the same resolution, so that eligibility is recomputed under the denser input. The results are reported below, in Table 2(f) and Table 34 of the revised paper, and in Section 4.4.

FramesElig.Base_E    ΔEGF
3260.774.3−13.275.8
6465.075.6−16.671.6

Denser sampling lowers the EGF rate by only 4.2 points, and every model still exceeds 65%. The failure therefore persists under denser sampling. Evidence removal still costs the models only 16.6 points of accuracy, against 64.4 points for humans.

Regarding the human comparison, human participants receive the same input modality as the models, namely the edited clips with aligned subtitles and no audio (now stated in Section 4.1 and Appendix B.3). Moreover, EGF compares each model with itself under a fixed frame budget, and the human reference only indicates whether the edited clip still supports a judgment. Humans who view the full frame rate with replays are the most likely to find any residual support, yet they fall to chance on these 353 events.

Minor (language): We have revised the text throughout to shorten long sentences, remove repeated qualifications in the captions of Tables 1 to 3, and unify terms such as EGF rate and label retention.

AuthorsOpenReview ↗

Gentle Reminder on Our Responses

Dear Reviewer H8bp,

Thank you again for your positive assessment and constructive comments. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.

Following your suggestions, we validated the EGF rate on the 353 events that humans judge undecidable after removal, where it remains 71.5%, against 71.8% on all paired events. We also added a 64-frame ablation, under which the EGF rate changes by only 4.2 points, and replaced Figure 1 with an event from this subset. These results are included in the revised paper (Tables 2(d) and 2(f), and Appendix O).

We would be happy to clarify any remaining questions.

Best regards, The Authors

I appreciate the authors' additional experiments, clarifications, and paper revisions. I would like to maintain my original score.

AuthorsOpenReview ↗

Thank You for Your Response

Thank you for your response and for your positive assessment of our work. We appreciate your acknowledgment of the additional experiments and revisions. Your suggestions on the events that humans judge undecidable and on denser frame sampling have strengthened the validation of the EGF rate, and these results will be kept in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses & Paper Revisions

We sincerely thank all the reviewers for their valuable and constructive reviews. We appreciate that the reviewers found the problem novel and underexplored (Reviewer H8bp), critical for social understanding in MLLMs (Reviewer 27yg), and clear and valuable as a research question (Reviewer p227). The reviewers also recognized the substantial annotation effort (Reviewer H8bp), the intervention design whose oracle hints localize the root cause of EGF (Reviewer 27yg), and the comprehensive experiments and controls (Reviewers H8bp, p227). Reviewer 27yg further found the paper well written with ample details.

We have made our best effort to address the concerns and revise the paper accordingly. All new results are collected in Appendix O (Tables 30 to 36). The main results also appear in new panels of the main text (Tables 2(d) to 2(f), Tables 3(c) and 3(d), and the Resid. column of Table 3(a)), and the key numbers are written into the corresponding sentences of Sections 3 to 6. The major modifications are summarized as follows.

  1. Validity of the EGF rate (Reviewers H8bp, p227): We report EGF on the 353 events where both human abstention responses to the evidence-removed version select cannot-be-determined (Table 2(d), Table 30, Section 4.3). Human accuracy on these events falls to 16.8% after removal, the chance level, yet the fifteen models still retain their labels on 71.5% of eligible events, against 71.8% on all paired events. Restricting EGF to correct intact judgments raises it to 77.5% (Table 31, Section 3.4). The EGF rate therefore does not overestimate grounding failure. Section 3.3 and Appendix A.2 now also state that events whose function is reconfirmed after the evidence interval were excluded during screening. Table 2(e) summarizes the EGF rate under all stricter conditions.

  2. Evidence perception and evidence use (Reviewers p227, H8bp): (1) On the baseline errors left unrepaired by combined target and interval hints, the local probe decodes the correct function in 55.5% of cases, against 17.2% for permuted labels (Tables 3(a), 3(d), and 32, Section 5.2). (2) Among 300 correctly localized errors under evidence-gated prompting, two raters judge the model's evidence description as matching the observed interaction in 66.3% of cases (Cohen's κ = 0.73). In 38.0% of these errors, the description matches and the probe decodes the correct function (Table 3(c), Tables 35 and 36, Section 5.3). (3) Repeating all L1, L2, and L3 queries with 64 frames lowers the EGF rate of the seven diagnostic models only from 75.8% to 71.6% (Table 2(f), Table 34, Section 4.4).

  3. Memorization (Reviewer 27yg): The macro EGF rate ranges only from 70.6% to 72.7% across the four source corpora (Table 33, Section 4.4), and Appendix O.4 discusses pretraining exposure.

  4. Figure 1 and presentation (Reviewer H8bp): Figure 1 now shows one of these 353 events, in which removing a fist bump leaves only the gaze and two participants standing still, and Section 4.3 describes the three model outputs. We have also revised the text throughout to shorten long sentences and reduce repeated qualifications. Section 4.1 and Appendix B.3 now state that human participants view the edited clips with aligned subtitles and no audio, the same modality as the models.

All major revised contents in the manuscript are highlighted in blue. Point-for-point responses to specific comments are given in the following reviewer-specific responses. We welcome any further discussions and will address any remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.