acceptodds
Withdrawn from ICLR 2027

Visual Evidence-based Reasoning for Multi-Object Multi-State Transitions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding object states in videos requires not only recognizing physical properties but also linking state changes to specific actions and conditions. Recognizing states and actions does not ensure correct object–state associations or causal explanations: Even when models correctly identify states, the explanations they provide may not align with the actual video content despite potentially conforming to common sense. Models therefore need to reason from visual evidence to connect each state change to the specific operations and conditions affecting the corresponding object. To address this, we propose a graph-guided visual evidence-based reasoning framework for understanding multi-object multi-state transitions in videos. A state graph is built based on timestamped segment descriptions and visual observations through iterative relation verification and state completion by leveraging a Vision-Language Model (VLM). In the graph, nodes represent objects and their states at specific stages, and edges represent object state transitions, multi-object combinations, or single-object separations. This structure links each state change to the corresponding object, relevant actions, and visual observations supporting these associations. For each transition, the VLM infers potential causes from object and contextual information in connected nodes and edges, accounting for implicit actions whose effects are difficult to observe locally. The timestamps retained in the graph are used to locate visual evidence and verify the consistency between state differences across nodes and actions represented by edges, thereby correcting erroneous object–state associations and revising or rejecting unsupported causal explanations. To support this reasoning framework, we construct VERMOST, a multi-object multi-state dataset, which comprises 1,827 third-person cooking videos and 75,977 annotated state transition events. Annotations include objects, pre- and post-transition states, a temporal interval, relevant actions, and specific causes along with their types. VERMOST links states and transitions to video-supported causes, covering explicit actions, implicit actions, and their joint effects. Experiments on VERMOST show that our framework outperforms the baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Object State Transitions, Causal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hVU9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VERMOST, a benchmark for multi-object, multi-state transition understanding that requires models to identify not only what changes in a video but also the specific causes supported by visual evidence. It further proposes a graph-guided reasoning framework that tracks object-state evolution, reasons over temporally distributed evidence, and verifies causal explanations, leading to substantial improvements over direct VLM prediction.

Critical strengths
  1. The figures are clear and easy to understand.
  2. The expeirments partially show the effectiveness of the proposed method.
Critical weaknesses
  1. The motivation is currently demonstrated mainly through failure cases of Qwen3.5-9B in Figure 1, making it unclear whether the identified object–state association and causal attribution errors remain substantial for stronger contemporary multimodal models.The paper should evaluate whether such failure modes persist with stronger backbones and whether the proposed graph-guided reasoning framework still provides consistent gains when applied to them. Otherwise, it is difficult to determine whether the proposed task addresses a general limitation of current video understanding systems or primarily compensates for weaknesses of the chosen backbone.
  2. The dataset construction pipeline raises a concern because the same Qwen3.5-9B model that is shown to suffer from object–state association and causal attribution errors is also used to generate the initial annotations. The paper should report the correction rate and human annotation cost, and ideally compare against independently annotated human labels to assess annotation bias.
  3. The Cause Score does not directly evaluate visual grounding, since the evaluator only compares the predicted and reference explanations without access to the video frames. This makes it unclear whether a high score truly reflects visual evidence-based reasoning rather than textual agreement with the reference explanation.
Questions for authors

See the weaknesses above.

Reviewer_PQ3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses multi-object, multi-state transition understanding in videos. Given a video, a model must find object-level state transitions and explain each one with a specific cause supported by the video, typed as explicit, implicit, or joint explicit–implicit. The proposed framework is training-free and runs entirely on Qwen3.5-9B with role-specific prompts. It first builds a state graph that links object observations across transformations, combinations, and separations. It then generates candidate causes for each change between pre- and post-transition states, selects or revises a cause, and verifies the events locally and across the whole video. The paper also introduces VERMOST, a dataset of 1,827 third-person cooking videos with 75,977 transition events, whose initial annotations were generated by Qwen3.5-9B and then reviewed by humans. On VERMOST, the framework outperforms several video LLMs, adapted baselines, and direct prediction with the same backbone, with the largest gains in event precision and in cause-type recognition for implicit and joint causes.

Critical strengths
  1. Well-motivated task. The paper targets a real gap: recognizing a state change is not enough, and the change must be linked to the correct object and to a cause supported by the video. The distinction between explicit, implicit, and joint causes is useful, because changes caused by heating, cooling, or evaporation are hard to explain from local observations alone.
  2. Large dataset with rich annotations. VERMOST provides 75,977 human-reviewed events over 205 hours of video. Each event has pre- and post-transition state sets, an interval, actions, tools, and a typed cause. This is substantially richer than existing state-change datasets (Table 1).
  3. Clear gains over the same backbone. Without any training, the framework improves on direct prediction with the same Qwen3.5-9B. Event F1 rises from 48.29 to 63.98, cause-type macro-F1 from 47.28 to 65.09, and the Cause Score from 7.17 to 8.21 (Table 2). For implicit and joint causes, the weighted type F1 rises from 22.16 to 61.05 (Table 3).
Critical weaknesses
  1. The same model is used for annotation, inference, evaluation, and baselines. Qwen3.5-9B generates the initial dataset annotations, runs every stage of the framework, scores the Cause Score, and is part of the adapted X-CLIP and OSCaR baselines. Even with human review, the references may be biased toward Qwen’s wording and granularity, which would favor Qwen-based methods. Consistent with this, the Qwen-based methods reach state F1 scores of 63–68, while the other video LLMs obtain 1–22 (Table 2). Since no other backbone or evaluator is tested, it is unclear whether the gains generalize.
  2. Unclear inputs and limited use of visual evidence. The paper does not explain how videos are segmented or which model produces the timestamped segment descriptions used as input. If these come from the same automatic segments used to create the initial annotations , the inputs and the references may overlap. Moreover, frames are used only during graph construction, while cause reasoning, selection, and verification operate on text alone . Speech transcripts are also used as context but are not mentioned in Section 3. These points weaken the claim of “visual evidence-based” reasoning.
  3. The evaluation protocol is underspecified and partly unvalidated. The Cause Score is computed by an LLM without video input, and its correlation with human judgments is not reported. The matching criteria (temporal overlap threshold and object correspondence), the matching of free-text states for the state F1 scores and Eq. 2, and the “weighted” metrics in Table 3 are not defined. No variance across runs or significance tests are reported.
  4. Concerns about the dataset as a benchmark. The paper does not report the reviewers’ number or expertise, inter-annotator agreement, or how much of Qwen’s initial annotations was changed during review. Implicit causes make up only 3.5% of events, which limits the reliability of the main claim about them. The dataset covers only cooking videos, and the whole dataset is used for evaluation without stating whether any part was used to design or tune the prompts.
  5. Unclear sources of improvement and limited novelty. Most of the gain over Qwen Direct comes from precision (41.55 → 78.06), while the state-difference F1 barely changes (62.87 → 63.24). Without the state graph, the framework performs almost the same as Qwen Direct (F1 48.98 vs. 48.29; Table 5), and the drop in recall when the cause-related stages are removed (54 → 32) is not explained. The improvement in Table 4 over Qwen Direct is also marginal (97.35 vs. 96.70). The framework is a prompt-based multi-stage pipeline with self-verification, but its relation to LLM self-refinement and graph-based video reasoning is not discussed, and its computational cost is not reported.
  6. Weak baselines. All video LLM baselines are 7B models, and their extremely low state F1 suggests an adaptation problem rather than a real capability gap. Larger or proprietary models are not evaluated. OSCaR was trained on egocentric data and sees only a composite of three frames, so it is poorly suited to third-person videos and to causes that occur well before the visible change.
Questions for authors
  1. How are videos segmented into the timestamped observations used for state graph construction, and which model generates the segment descriptions (Fig. 2(a), L216–217)? Are these the same automatic segments and descriptions that served as initial annotations before human review (L772–775, L950)? If so, how do the authors rule out information leakage from the references? (could change my assessment)
  2. Does the framework improve over direct prediction with other backbones (e.g., larger Qwen models, GPT-, or Gemini-class VLMs)? Can the authors report the Cause Score with a different evaluator, and its correlation with human judgments on a sample of events? (could change my assessment)
  3. How many reviewers annotated VERMOST, and what was their expertise and inter-annotator agreement? What fraction of Qwen’s initial annotations was corrected, added, or removed during review? Was any part of VERMOST used to design or tune the prompts? (could change my assessment)
  4. How much does visual input actually contribute, given that frames are used only in graph construction (L725–727) and speech transcripts are used as context (L740–741)? Can the authors report ablations that remove the frames or the speech transcripts? (could change my assessment)
  5. Please specify the matching protocol: the temporal overlap threshold, how object correspondence is determined, and how free-text states are normalized and matched for the state F1 scores and Eq. 2. How are the “weighted” metrics in Table 3 defined? Please also check that the implicit and joint Cause Scores of the full framework (both 7.88 in Table 5) and the weighted Cause Score in Table 3 (7.88) are correct. (clarification)
  6. Why does removing candidate reasoning or structured context reduce event recall from 54 to about 32 (Table 5)? What is the computational cost of the framework (number of VLM calls, tokens, runtime) compared with Qwen Direct? How does Qwen Direct perform with a comparable budget, e.g., with self-consistency or simple self-verification? (clarification)
Reviewer_th9kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VERMOST, a dataset of 1,827 cooking videos with 75,977 object-level state transitions, and a graph-guided framework for identifying and explaining these transitions. The framework uses a frozen VLM to construct state graphs, generate candidate causes, and check consistency. Results show improved event detection and causal attribution over direct prediction.

Critical strengths
  • The task is well motivated. Distinguishing visible manipulation from implicit effects, such as heat transfer, addresses a meaningful limitation of action recognition.
  • The state graph is a sensible design for maintaining object correspondence as ingredients change, combine, or separate. It also retains earlier observations that may explain later state changes.
  • The dataset provides detailed annotations connecting objects, coexisting states, and causes. The improvement over the same-backbone baseline is encouraging, particularly for event precision and implicit-cause attribution.
Critical weaknesses
  • The evidence for visual grounding is limited. Candidate reasoning and cause selection rely on recorded descriptions, while the Cause Score measures agreement with references without viewing the video. These checks may preserve errors in the initial descriptions rather than establish that explanations are visually supported.
  • Qwen3.5-9B generates initial annotations, performs inference, and evaluates explanations. Human review helps, but annotation agreement and independent validation of the evaluator are not reported.
  • The ablations do not cleanly isolate the proposed components. Removing the graph also removes state completion, and inference budgets are not matched. It is therefore difficult to separate the benefit of graph structure from additional context and computation.
  • The framework is evaluated only with Qwen3.5-9B. Applying it to another VLM would help establish whether the gains extend beyond this particular backbone.
Questions for authors
  1. Do the explanation gains hold under independent human evaluation against the videos, including checks for unsupported details?
  2. How much benefit remains against a non-graph baseline with the same observations, state completion, and inference budget?
  3. Can you report annotation agreement and provide the missing matching and inference details?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.