acceptodds
Withdrawn from ICLR 2027

Relation Event Measure: Preserving Event Supervision under Partial Grounding for Panoptic Video Scene Graph Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Panoptic video scene graph generation (PVSG) commonly constructs relation targets by matching annotations to predicted tracklets. When tracklets are incomplete, this matching process can discard entire events or truncate their positive temporal supervision. We introduce Relation Event Measure (REM), a mass-based formulation that decouples the full event annotation from partial visual groundings. To achieve this goal, we build a Relation Event Measure Prediction to separates event count and temporal allocation from grounding availability and tracklet-pair allocation. It preserves supervision for observed mass on matched pairs and unobserved mass in an explicit unobserved state. To supervise relation learning under REM, we propose the Annotation-Preserving Event Learning, which preserves event coverage and annotated temporal extent through measure supervision over all valid events and full-span temporal supervision. Furthermore, we develop the REM-based Relation Inference that predicts relations with integrated temporal mass and retained temporal localization. REM improves strict mR@100 over retained-event supervision by 3.43 and 0.37 percentage points under IPS+T and VPS, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
panoptic video scene graph generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YB2XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the loss or truncation of relation supervision caused by incomplete predicted tracklets in panoptic video scene graph generation (PVSG). The authors find that conventional matching pipelines discard 12.2% and 35.5% of training events under the IPS+T and VPS settings, respectively. Motivated by this observation, they propose Relation Event Measure (REM), which represents each complete annotated event as normalized temporal mass and factorizes predictions into event count, temporal allocation, grounding availability, and tracklet-pair allocation. An explicit unobserved state preserves supervision that cannot be grounded. Training combines measure supervision over all valid events with full-span temporal supervision for events with available correspondences. At inference, integrated mass supports relation scoring, while mass-weighted queries support temporal localization. On PVSG, compared with an architecture-matched control trained only on retained events, REM improves strict mR@100 from 10.85 to 14.28 under IPS+T and from 2.44 to 2.81 under VPS.

Critical strengths
  1. A clearly identified, practically relevant problem. The paper quantifies the loss of existing annotations caused by matching-based supervision construction and highlights that detection or tracking failures should not automatically lead to discarded relation supervision. This provides a clear motivation for the work.
  2. A well-motivated supervision design. REM separates event existence and timing from visual grounding, preserving supervision through an unobserved state without assigning unsupported object identities. Training and inference use a unified mass representation.
Critical weaknesses
  1. It remains unclear whether the gains depend on relatively weak upstream detection and tracking. The experiments use incomplete predicted tracklets provided by the benchmark, without validation on higher-performing detection and tracking systems. As upstream performance improves and matching-induced supervision loss decreases, would REM's incremental benefit also diminish? Could simply upgrading the upstream model yield a larger improvement than adopting REM? Without such comparisons, the method's practical value and complementarity with upstream improvements remain unclear.
  2. Some gains are small, with limited evidence of robustness. Under VPS, the improvement in strict mR@100 over SegPVSG without augmentation is only 0.02 percentage points. The main results do not report standard deviations or confidence intervals, making it difficult to assess the consistency of such small gains.
  3. The comparison with the full strong baseline remains insufficient. The main table uses SegPVSG without RGVA augmentation, whereas the appendix shows that full SegPVSG achieves higher strict mR@100 than REM under both settings. Although augmentation introduces additional training data, REM's advantages relative to the full method and its complementarity with augmentation need clearer characterization.
  4. Validation of generalization and temporal localization remains limited. Experiments focus on a single small dataset, leaving it unclear whether the gains persist across other datasets or different upstream tracklet quality levels. Scores under the stricter temporal-localization criterion remain low, suggesting that preserving full-span supervision has not yet fully translated into accurate event-boundary prediction.
Questions for authors

See "Weakness".

Reviewer_MGqBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the truncation and deletion of relation supervision caused by matching annotations to predicted tracklets in panoptic video scene graph generation. The proposed Relation Event Measure (REM) represents each annotated event as normalized temporal mass, factorized into event count, temporal allocation, observability, and pair allocation. Unmatched mass is assigned to an explicit unobserved state, allowing events discarded by conventional preprocessing to retain partial supervision. Events with valid correspondences additionally train an interval decoder using their complete temporal annotations. Experiments are conducted on PVSG under IPS+T and VPS, with retained-event controls, BCE baselines, and scoring ablations.

Critical strengths
  1. The paper identifies a practical supervision-construction issue and quantifies its prevalence: 12.2% and 35.5% of valid training events are discarded under IPS+T and VPS, respectively, with short events disproportionately affected. This provides a reasonable motivation for revisiting annotation utilization during tracklet matching.

  2. The evaluation diagnostics are useful. The authors identify that the released metric can reward overly long relation predictions and supplement it with temporal-IoU-gated metrics and localization analysis on common candidates. These analyses help distinguish retrieval improvements from better temporal localization.

Critical weaknesses
  1. The main comparison does not isolate the contribution of event-mass modeling.

The main text emphasizes REM’s 4.74-point IPS+T mR@100 improvement over Simple BCE. However, this comparison also changes observability supervision, loss weighting, query construction, and inference scoring. It therefore does not isolate the contribution of REM’s mass-allocation mechanism. More carefully matched controls appear only in Appendix D and should receive greater prominence.

Under the common softmax readout and matched temporal weights, the IPS+T/Q CE–BCE difference is only 0.953 points, with a 95% confidence interval of [−2.089, 2.358]. Although IPS+T/G shows a significant gain, most other matched contrasts are small or inconclusive. The evidence therefore supports configuration-specific gains rather than a consistent advantage of REM.

  1. The empirical gains are limited in scope and lack training-seed uncertainty.

The evaluation is limited to PVSG under two fixed upstream observation settings. Under VPS, strict mR@100 improves only from 2.79 for SegPVSG without generative augmentation to 2.81 for REM. Full SegPVSG reaches 4.82, although it uses additional synthetic augmentation and is not a supervision-matched baseline.

The main results average three runs, but individual results and dispersion are not reported. The appendix bootstrap conditions on fixed fitted models and does not measure training-seed variability. This is especially important for the small VPS improvements. The experiments demonstrate value in restoring discarded annotations, but do not establish that the gains are stable or broadly generalizable.

  1. The additional methodological value of REM beyond annotation retention remains unclear.

The clearest contribution is preserving annotations that would otherwise be deleted during tracklet matching. REM additionally introduces event-count prediction, hierarchical normalized allocation, and an explicit unobserved state. However, the paper does not clearly establish what these components add beyond retaining all events and complete temporal labels with matched count, temporal, and observability supervision.

The stronger appendix controls show that the advantage depends on the loss, scoring rule, and evaluation metric. The current evidence therefore supports annotation retention more strongly than it supports the necessity of REM’s specific formulation.

  1. Proposition 1 has limited relevance to practical tracklet fragmentation.

Proposition 1 fixes observations, features, and correspondences, changes only tracklet indexing, and then reaggregates the resulting statistics. This property is mathematically valid but largely follows from set-encoder equivariance and additive aggregation. It does not establish robustness to missed detections, identity switches, or unrecombined fragments at inference.

This limitation matters because training queries can aggregate fragments using ground-truth correspondences, whereas inference operates on native predicted tracklet pairs. A controlled experiment that progressively fragments tracklets without ground-truth merging at inference would better demonstrate the practical relevance of the claimed property.

Questions for authors
  1. Why is Simple BCE the principal control in the main text, while the more carefully matched baselines in Appendix D are relegated to the appendix? After matching global supervision, query construction, and training budgets, and using reasonable scoring rules for both methods, which gains remain specifically attributable to REM’s mass-allocation mechanism?

  2. Can the authors provide per-seed results and paired differences for REM, Retained REM, and the strongest matched BCE baseline? In particular, are the small VPS improvements consistent across training seeds?

  3. Relative to retaining all events and complete temporal labels with count and temporal auxiliary supervision, which component of REM provides an additional benefit that these simpler alternatives cannot achieve? Can the existing ablations separate this benefit from general supervision recovery?

  4. How does Proposition 1 support the practical inference setting? With observations fixed, how does performance change as tracklets are increasingly fragmented without ground-truth-based merging at inference? Can this analysis clarify the mismatch between training and inference queries?

Reviewer_LzU5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of supervision loss in Panoptic Video Scene Graph Generation (PVSG) caused by tracklet-level matching, where annotated relation events are discarded or truncated when predicted tracklets fail to provide valid subject-object correspondences. The authors propose Relation Event Measure (REM), which assigns each annotated event a unit of temporal mass distributed across observed subject-object pairs and an explicit unobserved state. All valid events contribute to measure supervision; events with at least one valid correspondence additionally train a temporal-span predictor. Experiments on PVSG under IPS+T and VPS settings show improvements over SegPVSG, MCL, and UNO baselines, with ablations examining event retention, query construction, and relation scoring.

Critical strengths
  1. Well-motivated problem with convincing controlled evidence. The paper identifies a significant supervision loss in PVSG, where tracklet matching discards 12.2% and 35.5% of annotated events under IPS+T and VPS, respectively, including events with valid sampled correspondences. The architecture-matched Retained REM baseline isolates the effect of restoring deleted events, demonstrating meaningful improvements in relation recall. Additional ablations further support the value of preserving otherwise discarded annotations.
  2. Thorough experimental analysis and transparent evaluation. The authors identify a limitation of the standard PVSG metric, which can reward overly long temporal predictions. Their analysis shows that extending predictions to the full video increases mR but decreases temporally gated G-mR. Additional evaluations, including matched CE/BCE comparisons, temporal localization diagnostics, and bootstrap confidence intervals, provide useful insights. The authors also acknowledge unfavorable results and limitations, strengthening the credibility of their empirical analysis. Critical Weakness
Critical weaknesses
  1. The VPS results are fragile and the paper’s own ablations undermine the claimed benefit. Table 3 shows that under VPS, adding all events (2.81 mR) performs worse than adding only events without sampled correspondences (2.84 mR), and uniform interval queries actually improve VPS mR to 3.08 while reducing G-mR. The authors acknowledge this non-monotonicity but do not resolve whether the full REM machinery is justified under VPS, or whether a simpler variant would suffice. This weakens the “one method fits both settings” narrative.

  2. Insufficient engagement with the fundamental limitation of VPS recoverability. Table 6 reports that under VPS, only 48/954 validation events are recoverable with fixed native tracklet pairs, yielding an mR ceiling of 8.01%. The reported VPS mR@100 of 2.81 (34.9% of ceiling) is not obviously impressive, yet the paper does not adequately discuss whether the method is approaching a meaningful fraction of what the fixed tracks can support, or whether the remaining gap is attributable to REM versus fundamental data limitations.

Questions for authors
  1. Table 6 shows that REM achieves only 2.81% mR@100 under VPS, compared with the fixed-tracklet recoverability ceiling of 8.01%. Could the authors further analyze the remaining gap by distinguishing failures in relation prediction, candidate ranking, and temporal localization among spatially recoverable events?

  2. Table 3 shows that uniform interval queries outperform the full REM configuration in VPS mR@100 (3.08 vs. 2.81), while achieving slightly lower G-mR@100 (0.54 vs. 0.56). Could the authors clarify whether mass-weighted queries provide a consistent and statistically reliable advantage over uniform queries? These results raise questions about the necessity of this component, particularly under sparse observations.

  3. The paper states that “the smaller VPS gain is consistent with its lower fixed-tracklet spatial recoverability.” However, if REM’s contribution is precisely to extract more supervision from incomplete observations, shouldn’t we expect a larger relative gain when observations are scarcer? Can the authors clarify the mechanism by which low recoverability suppresses REM’s advantage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.