acceptodds
Desk rejected at ICLR 2027

THG-Omni: Traceable Human-Centric Graph Supervision for Video Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Omni-modal models can answer questions about video, but their predictions rarely show which participant, event, or interval supports an answer. Moreover, answer-level supervision alone does not teach a model to interpret structured video evidence. To systematically mitigate this issue, we present THG-Omni, a human-centric graph-supervision framework that links Segment Anything Model 3 participant tracks with temporally grounded Qwen3-VL annotations. Candidate semantic claims are checked against the source video and available audio before they are used to construct training questions, while track geometry provides explicit spatial and temporal supervision. The resulting questions teach an omni-modal reasoning model to interpret participant attributes, actions, affect, interactions, and their timing from a structured graph. At inference, THG-Omni uses a predicted graph constructed from the evaluation video. Experiments across social reasoning, emotion understanding, and deception perception show consistent improvements over the unadapted backbone. A complementary diagnostic demonstrates that the predicted graph preserves evidence useful for answering questions even without the source video and audio. Together, these findings show that THG-Omni improves human-centric video reasoning while grounding its evidence in identifiable participants and events. Our codes are available at https://anonymous.4open.science/r/review-n94hhfcwp5666/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Centric Video Reasoning, Video Scene Graphs, Multimodal Large Language Models, Temporal Grounding, Multimodal Video Understanding, Structured Visual Reasoning

Related papers

Discussion (0)

Sign in to comment.