- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the loss or truncation of relation supervision caused by incomplete predicted tracklets in panoptic video scene graph generation (PVSG). The authors find that conventional matching pipelines discard 12.2% and 35.5% of training events under the IPS+T and VPS settings, respectively. Motivated by this observation, they propose Relation Event Measure (REM), which represents each complete annotated event as normalized temporal mass and factorizes predictions into event count, temporal allocation, grounding availability, and tracklet-pair allocation. An explicit unobserved state preserves supervision that cannot be grounded. Training combines measure supervision over all valid events with full-span temporal supervision for events with available correspondences. At inference, integrated mass supports relation scoring, while mass-weighted queries support temporal localization. On PVSG, compared with an architecture-matched control trained only on retained events, REM improves strict mR@100 from 10.85 to 14.28 under IPS+T and from 2.44 to 2.81 under VPS.
Critical strengths
- A clearly identified, practically relevant problem. The paper quantifies the loss of existing annotations caused by matching-based supervision construction and highlights that detection or tracking failures should not automatically lead to discarded relation supervision. This provides a clear motivation for the work.
- A well-motivated supervision design. REM separates event existence and timing from visual grounding, preserving supervision through an unobserved state without assigning unsupported object identities. Training and inference use a unified mass representation.
Critical weaknesses
- It remains unclear whether the gains depend on relatively weak upstream detection and tracking. The experiments use incomplete predicted tracklets provided by the benchmark, without validation on higher-performing detection and tracking systems. As upstream performance improves and matching-induced supervision loss decreases, would REM's incremental benefit also diminish? Could simply upgrading the upstream model yield a larger improvement than adopting REM? Without such comparisons, the method's practical value and complementarity with upstream improvements remain unclear.
- Some gains are small, with limited evidence of robustness. Under VPS, the improvement in strict mR@100 over SegPVSG without augmentation is only 0.02 percentage points. The main results do not report standard deviations or confidence intervals, making it difficult to assess the consistency of such small gains.
- The comparison with the full strong baseline remains insufficient. The main table uses SegPVSG without RGVA augmentation, whereas the appendix shows that full SegPVSG achieves higher strict mR@100 than REM under both settings. Although augmentation introduces additional training data, REM's advantages relative to the full method and its complementarity with augmentation need clearer characterization.
- Validation of generalization and temporal localization remains limited. Experiments focus on a single small dataset, leaving it unclear whether the gains persist across other datasets or different upstream tracklet quality levels. Scores under the stricter temporal-localization criterion remain low, suggesting that preserving full-span supervision has not yet fully translated into accurate event-boundary prediction.
Questions for authors
See "Weakness".