- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that frame-level training and evaluation in Video Scene Graph Generation (VSGG) induce duration bias by assigning greater weight to relation events that last for more frames, thereby underrepresenting many short events. The authors introduce the event-aware metric ER@K and derive a decomposition connecting frame-level Recall to ER@K. They further propose Kairos, which uses an event-balanced loss and an event clock to mitigate duration bias.
Critical strengths
The core insight is clear and substantive. In Video Scene Graph Generation VSGG, Recall is computed at the frame level over subject–predicate–object triplets. In practice, the relation between a pair of entities remains unchanged far more often than it changes, meaning that long-duration relations are repeatedly counted, as the authors point out. Therefore, relying on Recall alone as the evaluation metric is clearly inadequate. The paper is thus motivated by a strong and well-founded observation. Moreover, the authors conduct extensive experiments on Action Genome, which is effectively the only widely established benchmark for VSGG.
Critical weaknesses
-
The provenance of the results across the five hosts in Table 3 is unclear. The caption of Table 3 states that the violet entries are “projected from the controlled study,” whereas the Abstract, Section 1, and Section 5.2 describe them as results of Kairos across five hosts. Section E of the supplementary material mainly explains how Kairos is integrated into different hosts, but does not provide the corresponding complete experimental settings or results from independent runs. If these numbers are indeed projected, they are insufficient to directly support the main claim that Kairos is effective across five different hosts.
-
The current evidence primarily supports the event-balanced loss rather than the complete event-clock design. Appendix D of the supplementary material states that the hazard head is trained after the transformer on frozen encodings and that “the relation branch of Kairos is exactly the transformer trained with the event-balanced loss.” Section E further states that the default configuration uses no readout and no attention bias. In Tables S7–S9, event-time attention provides only small gains and is slightly worse in some settings. Table S10 shows that event-time readout can be useful, but it introduces a trade-off between R@K and ER@K. Therefore, the paper's presentation of the complete Kairos framework is currently stronger than what the experiments directly support.
-
Some short events may result from annotation flicker. Section 3.1 reports that approximately 74% of interior single-frame events return to the interrupted relation, and Section 6 acknowledges that some of these cases may arise from annotation flicker. Appendix C of the supplementary material further states that, in some cases, the labels alone cannot distinguish a genuine short action from annotation noise. Since the main gains are concentrated on short events, further evidence is needed to establish that the improvements do not mainly come from fitting such annotation flicker.
Questions for authors
-
I suggest revising the “one unit of time per frame/event” illustration in Figure 1. The current visualization may intuitively give the impression that predictions become easier simply because event-level evaluation is used, rather than clarifying that clock time and event time apply different counting and weighting schemes.
-
I suggest adding several concrete and representative examples that compare how R@K and ER@K evaluate the same predictions. Such cases would demonstrate the practical meaning of ER@K more directly than the aggregate distributions alone.
3 comments
Response to Reviewer 7UC5 (Part 2/3)
Weakness 2: Design of the Event Clock
Thank you for the in-depth examination of our event clock design. Kairos brings event time to VSGG through two complementary parts, each with its own role: the event-balanced loss corrects the duration bias in training, and the event clock extends event time beyond training, to analysis and inference (Sec. 4.4). We clarify the role of the clock in three points:
① The small effect of event-time attention is the finding itself, not a missing gain. It is a diagnostic experiment designed to locate where the duration bias enters a temporal model, in its temporal aggregation or in its training objective (Sec. 5.3): moving attention to event time, all else fixed, changes the overall metrics by at most half a point (Table 4, Tables S7–S9), whereas changing how training counts events raises ER1 by 1.7 at unchanged R@10. The duration bias therefore stems from the counting unit of the training objective (frames instead of events), not from the temporal aggregation mechanism, as stated in contribution (iv) of Sec. 1 (L122–125).
② The clock is decoupled from the relation branch by design. The hazard head is trained on frozen encodings (App. D), because training it jointly makes it no more accurate but costs 0.9 points of tail event recall (Table S8). This decoupling makes the clock a plug-in that can be added to any trained model without retraining and without risk to its predictions. Supervised by relation changes, it detects them with an AUROC of 73.3, against 56.4 without supervision (Table 6).
③ Once switched on, the clock does change the predictions. As a readout, it corrects an already trained model without retraining, raising ER@10 by 1.0 for 0.3 points of R@10, about 4 times as much as a class-level duration prior, and moves the α=0.5 model to the α=1 operating point (74.6/59.3 vs. 74.7/59.1, Table S10). We keep it off by default so that frame-level recall stays intact and the trade-off is left to the application. This gain cannot come from reweighting, because the weights are fixed once training ends.
In summary, each part of the Kairos framework is backed by direct evidence. As our main contribution, we reveal the duration bias of the traditional frame-level paradigm, support it with sufficient statistics (Fig. 1b–c, Eq. 1), and design Kairos accordingly, which provides the community with a better event-centric paradigm for training, evaluation and optimization. As a minor contribution, we verify its generalization ability through extensive experiments on four representative types of baselines (one-stage, two-stage, traditional debiasing and MLLM-based) under settings of different difficulty, PredCls and SGDet (Table 3). The event clock completes the framework as a decoupled component with its own measured benefits (Table 6, Table S10).
Response to Reviewer 7UC5 (Part 3/3)
Weakness 3: Annotation Flicker and Event Definition
Thank you for the valuable comment. We examined this question from three angles, and all of them indicate that the gains of Kairos come mainly from genuine short relations:
① The gains concentrate on genuine brief actions. The two predicates that gain most are eating (+10.3) and drinking from (+5.0, Table S6). For them, returning to the interrupted relation is the natural course of the action (e.g., holding, drinking, holding): 87% and 91% of their interior single-frame events do so, against 74% overall (L164–166). We further checked such cases against the video: every drinking event in the real clips of the new Fig. S2 (App. C) is a real sip, and the single frame of touching that Kairos recovers in Fig. 4 marks putting the object down or picking it up (L482–485).
② The gains follow duration and are not confined to single-frame events. Events of two to three frames, whose labels persist across consecutive keyframes, gain 3.0 points of ER@10 (56.9→59.9), close to the 3.5 points of single-frame events (42.8→46.3), while longer events stay in place (α=1, Fig. 3a). Across predicates, the gain falls with the mean duration (Spearman ρ=−0.73, Fig. 3b).
③ The conclusions hold under both event definitions used in the paper, and the analysis carries over to the suggested merge. The gains hold whether events are broken where the object disappears (App. B; ER1 +1.1 and ERtail +1.9 on average, Table 3) or merged across such gaps (App. D; ER1 +1.7 at unchanged R@10, Table 4). For the suggested merge, the identity of Eq. 1 holds for any definition of an event: the interrupted relation becomes one longer event, while the brief event that interrupts it remains a single-frame event, so the gap in duration behind the bias widens and the event-balanced loss still shifts weight to the brief event.
In summary, the three results consistently indicate that the gains of Kairos mainly reflect genuine brief relations. The real clips of Fig. S2 are added to App. C in the revised PDF. We hope this also answers your question on merging one-frame interruptions.
Questions 1–2: Illustration in Figure 1 and Concrete Examples
Thank you for both suggestions, which make the meaning of event time and ER@K clearer. Both are adopted in the revised PDF:
① Figure 1(a) now shows clock time and event time as two weighting schemes over the same frames and labels. The two bars count the same seven frames, and the width of each segment is its share of the weight: the drink carries 1/7 with one unit of time per frame and 1/3 with one unit per event. The outcome on the right is marked "trained with these weights": the drink is recovered because the model is trained with event weights, not because the evaluation is easier. For the same prediction, ER@K is in fact the stricter score: the clock-time transformer reaches 75.4 in R@10 but 56.9 in ER@10 (Table 4).
② Real examples that compare R@K and ER@K on the same predictions are added (Fig. S2, App. C). In three real Action Genome test clips, a person drinks from a cup between spells of holding it. The same prediction is scored on every clip: holding is recalled in every keyframe and drinking from is never reported. R@K stays at 76.9–86.2 (e.g., 10 of 13 labels), whereas ER@K is 33.3 on every clip, since the two missed sips are two of the three events. Similarly, on a real test track (Fig. 4a), the frame of touching that the clock-time transformer skips is one of 12 contact triplets but one of three events (ER 0.67, against 1.00 for Kairos; L482–485).
These examples show the practical meaning of ER@K: it asks whether each relation instance is found, however brief.
Thank you again for the insightful comments. If there are any remaining questions, we are more than happy to address them.
Response to Reviewer 7UC5 (Part 1/3)
We sincerely thank Reviewer 7UC5 for the valuable comments. We address each concern below, and all revisions are marked in red in the updated PDF.
Since the settings of VSGG are complex and the representative methods are numerous, we did not report all baselines at submission. In the updated PDF, we have completed all direct experimental results, covering four representative types of methods: one-stage, two-stage, traditional debiasing and MLLM-based. Each method is trained and evaluated with its released code and the same evaluation, under two task settings that range from simple to complex, PredCls and SGDet. Recall is the traditional frame-level metric; event recall (ER), single-frame event recall (ER1) and tail event recall (ERtail) are our event-level metrics (K=20; ↑/↓: change over the base model):
Every result supports our main claim, which holds on each type of base model: after adding Kairos, the traditional Recall stays unchanged (within 0.2 points), while all of our event-level metrics improve. On average, ER, ER1 and ERtail rise by 0.6, 1.1 and 1.9 points, the last being 17% of the tail's event recall. We reveal the underlying duration bias of the frame-level paradigm, support it with sufficient statistics, and design Kairos accordingly, moving existing VSGG from the frame level to the event level in training, evaluation and optimization. Together with the above experiments on generalization ability, this provides sufficient support for applications that are sensitive to short relations.