- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed Scene Evolution Understanding (SEU): from observed video, a model must track entities, predict their future trajectories and events, point to supporting Visual Evidence in the observed interval, and describe the scene. The paper introduces a new SceneForesee benchmark from APIDIS, OVT-B and TAO, with Events derived from trajectory geometry and descriptions/evidence generated with LLM assistance and manual inspection. It also proposes EvoSee, a query-based model with evidence encoding, multi-hypothesis future prediction where geometry and event heads share candidate states and are trained by joint assignment, and a gated refinement that feeds the selected continuation back into entity and scene representations. EvoSee outperforms tracking adaptations and zero-/few-shot LMMs on SceneForesee (e.g., +10.45 Combined Point-F1 and +17.24 Event Global F1 over Qwen3.5-9B) and improves Dyn-Bench VQA (+4.8) and grounding J&F (+3.58) over its base systems.
Critical strengths
- Using persistent identities to tie observed evidence, future trajectories and events together, with joint assignment over shared candidate states, is a coherent design. The ablations (geometry-only assignment, single hypothesis, observation-only refinement, no visual evidence) consistently show that each coupling contributes, so the model's internal claims are reasonably supported.
- The Observation-only Refinement control on Dyn-Bench keeps the architecture fixed and replaces predicted change with observed summaries. It still shows gains, which suggests that forecasting provides useful context beyond extra parameters.
Critical weaknesses
- Approach, occlusion and leave are defined by box separation, overlap and edge proximity, and relation change is the residual. Event prediction is therefore largely re-encoded trajectory forecasting with hand-set thresholds, not semantic understanding. Since EvoSee is trained on exactly these heuristics while the LMM baselines (Qwen, InternVL, LLaVA) appear to be prompted without task training, the large Event/Evidence gains mostly reflect fitting the label definitions. Meanwhile, BERTScore for descriptions is essentially tied.
- Descriptions and evidence are partly LLM-generated ("synthetic data generation" in the AI-use statement), with unspecified "LLM assessment and manual inspection". No inter-annotator agreement, error rate, or size of the human-verified subset is reported. Results are single runs with no variance, and absolute continuity scores are low (~27–31 Point-F1), so it is unclear how much of the remaining error is label noise.
Questions for authors
- Are any of the LMM baselines fine-tuned on SceneForesee? If not, please report at least one LMM (e.g., Qwen3.5-9B, which EvoSee seems to build on) fine-tuned on the same training split, so that the gain from the EvoSee architecture is separated from task-specific training.
- How accurate are the Event labels against human judgment? What fraction of the benchmark was manually verified, and what was the disagreement rate? (more of a future work)
- What happens if a simple rule applied to EvoSee's (or a tracker's) predicted future boxes is used to produce Events? If this matches the learned Event head, the semantic branch adds little beyond geometry.
- Please report variance across training seeds for Tables 1–2 and 8–9, particularly for the smaller Dyn-Bench gains (e.g., +0.08 on Object).