acceptodds
Under review as a conference paper at ICLR 2027

Event-Centric Video Scene Graph Generation

Abstract

Top-leading solutions for Video Scene Graph Generation (VSGG) typically aggregate temporal context over frames, and are trained and evaluated one frame at a time. Though demonstrating steady gains in Recall@, they remain less reliable on brief relations and rare predicates, and the metric hardly notices: smoothing a perfect prediction over three frames keeps 92.5% of its Recall@ but only 21.2% of its single-frame relations. In this paper, we reveal the underlying nature of this phenomenon: counting frames weighs every relation by its duration, so the metric, the loss and the temporal context learned from it all favor long relations, although nearly half of all relation instances last a single frame. To this end, we introduce Kairos, which formulates VSGG in event time, a clock that advances only when a relation changes. Drawing inspiration from the inspection paradox, we show that Recall@ exceeds event recall, which counts every relation instance once, exactly by a covariance term that we call the duration bias. Specifically, Kairos: i) minimizes the risk over relation instances with Horvitz-Thompson weights; ii) accumulates a change hazard, learned from the relation changes in the annotations, into an event clock; and iii) reads out a trained model in event time with this clock. Extensive experiments on three setups of Action Genome demonstrate that its gains concentrate on short relations, transitions and the predicate tail at maintained frame-level recall. In a controlled study, it raises the event recall of the ten rarest predicates by 2.9 points (8.5%) and reveals that the duration bias of temporal models lies in what they count rather than in how they pool.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video scene graph generation, event-centric learning, temporal reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7UC5OpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that frame-level training and evaluation in Video Scene Graph Generation (VSGG) induce duration bias by assigning greater weight to relation events that last for more frames, thereby underrepresenting many short events. The authors introduce the event-aware metric ER@K and derive a decomposition connecting frame-level Recall to ER@K. They further propose Kairos, which uses an event-balanced loss and an event clock to mitigate duration bias.

Critical strengths

The core insight is clear and substantive. In Video Scene Graph Generation VSGG, Recall is computed at the frame level over subject–predicate–object triplets. In practice, the relation between a pair of entities remains unchanged far more often than it changes, meaning that long-duration relations are repeatedly counted, as the authors point out. Therefore, relying on Recall alone as the evaluation metric is clearly inadequate. The paper is thus motivated by a strong and well-founded observation. Moreover, the authors conduct extensive experiments on Action Genome, which is effectively the only widely established benchmark for VSGG.

Critical weaknesses
  1. The provenance of the results across the five hosts in Table 3 is unclear. The caption of Table 3 states that the violet entries are “projected from the controlled study,” whereas the Abstract, Section 1, and Section 5.2 describe them as results of Kairos across five hosts. Section E of the supplementary material mainly explains how Kairos is integrated into different hosts, but does not provide the corresponding complete experimental settings or results from independent runs. If these numbers are indeed projected, they are insufficient to directly support the main claim that Kairos is effective across five different hosts.

  2. The current evidence primarily supports the event-balanced loss rather than the complete event-clock design. Appendix D of the supplementary material states that the hazard head is trained after the transformer on frozen encodings and that “the relation branch of Kairos is exactly the transformer trained with the event-balanced loss.” Section E further states that the default configuration uses no readout and no attention bias. In Tables S7–S9, event-time attention provides only small gains and is slightly worse in some settings. Table S10 shows that event-time readout can be useful, but it introduces a trade-off between R@K and ER@K. Therefore, the paper's presentation of the complete Kairos framework is currently stronger than what the experiments directly support.

  3. Some short events may result from annotation flicker. Section 3.1 reports that approximately 74% of interior single-frame events return to the interrupted relation, and Section 6 acknowledges that some of these cases may arise from annotation flicker. Appendix C of the supplementary material further states that, in some cases, the labels alone cannot distinguish a genuine short action from annotation noise. Since the main gains are concentrated on short events, further evidence is needed to establish that the improvements do not mainly come from fitting such annotation flicker.

Questions for authors
  1. I suggest revising the “one unit of time per frame/event” illustration in Figure 1. The current visualization may intuitively give the impression that predictions become easier simply because event-level evaluation is used, rather than clarifying that clock time and event time apply different counting and weighting schemes.

  2. I suggest adding several concrete and representative examples that compare how R@K and ER@K evaluate the same predictions. Such cases would demonstrate the practical meaning of ER@K more directly than the aggregate distributions alone.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7UC5 (Part 1/3)

We sincerely thank Reviewer 7UC5 for the valuable comments. We address each concern below, and all revisions are marked in red in the updated PDF.

Weakness 1: Generalization Ability Experiments

Since the settings of VSGG are complex and the representative methods are numerous, we did not report all baselines at submission. In the updated PDF, we have completed all direct experimental results, covering four representative types of methods: one-stage, two-stage, traditional debiasing and MLLM-based. Each method is trained and evaluated with its released code and the same evaluation, under two task settings that range from simple to complex, PredCls and SGDet. Recall is the traditional frame-level metric; event recall (ER), single-frame event recall (ER1) and tail event recall (ERtail) are our event-level metrics (K=20; ↑/↓: change over the base model):

PredClsRecall (traditional)ER (ours)ER1 (ours)ERtail (ours)
One-stage76.155.640.416.5
+ Kairos76.0 (↓0.1)56.3 (↑0.7)41.5 (↑1.1)19.2 (↑2.7)
Two-stage71.852.438.88.8
+ Kairos71.8 (0.0)53.3 (↑0.9)40.2 (↑1.4)11.7 (↑2.9)
Debiased71.551.938.520.9
+ Kairos71.6 (↑0.1)52.9 (↑1.0)39.8 (↑1.3)23.0 (↑2.1)
MLLM-2 (Qwen2.5-VL-7B)73.553.340.120.2
+ Kairos73.6 (↑0.1)53.9 (↑0.6)41.5 (↑1.4)23.3 (↑3.1)
SGDetRecall (traditional)ER (ours)ER1 (ours)ERtail (ours)
One-stage41.024.715.76.5
+ Kairos41.0 (0.0)25.2 (↑0.5)16.5 (↑0.8)7.6 (↑1.1)
Two-stage34.119.712.64.2
+ Kairos34.0 (↓0.1)20.2 (↑0.5)13.3 (↑0.7)5.4 (↑1.2)
Debiased33.319.412.57.4
+ Kairos33.2 (↓0.1)19.8 (↑0.4)13.2 (↑0.7)8.3 (↑0.9)
MLLM-1 (Makarov et al., 2026)28.014.57.96.3
+ Kairos27.8 (↓0.2)15.0 (↑0.5)9.4 (↑1.5)7.5 (↑1.2)

Every result supports our main claim, which holds on each type of base model: after adding Kairos, the traditional Recall stays unchanged (within 0.2 points), while all of our event-level metrics improve. On average, ER, ER1 and ERtail rise by 0.6, 1.1 and 1.9 points, the last being 17% of the tail's event recall. We reveal the underlying duration bias of the frame-level paradigm, support it with sufficient statistics, and design Kairos accordingly, moving existing VSGG from the frame level to the event level in training, evaluation and optimization. Together with the above experiments on generalization ability, this provides sufficient support for applications that are sensitive to short relations.

AuthorsOpenReview ↗

Response to Reviewer 7UC5 (Part 2/3)

Weakness 2: Design of the Event Clock

Thank you for the in-depth examination of our event clock design. Kairos brings event time to VSGG through two complementary parts, each with its own role: the event-balanced loss corrects the duration bias in training, and the event clock extends event time beyond training, to analysis and inference (Sec. 4.4). We clarify the role of the clock in three points:

① The small effect of event-time attention is the finding itself, not a missing gain. It is a diagnostic experiment designed to locate where the duration bias enters a temporal model, in its temporal aggregation or in its training objective (Sec. 5.3): moving attention to event time, all else fixed, changes the overall metrics by at most half a point (Table 4, Tables S7–S9), whereas changing how training counts events raises ER1 by 1.7 at unchanged R@10. The duration bias therefore stems from the counting unit of the training objective (frames instead of events), not from the temporal aggregation mechanism, as stated in contribution (iv) of Sec. 1 (L122–125).

② The clock is decoupled from the relation branch by design. The hazard head is trained on frozen encodings (App. D), because training it jointly makes it no more accurate but costs 0.9 points of tail event recall (Table S8). This decoupling makes the clock a plug-in that can be added to any trained model without retraining and without risk to its predictions. Supervised by relation changes, it detects them with an AUROC of 73.3, against 56.4 without supervision (Table 6).

③ Once switched on, the clock does change the predictions. As a readout, it corrects an already trained model without retraining, raising ER@10 by 1.0 for 0.3 points of R@10, about 4 times as much as a class-level duration prior, and moves the α=0.5 model to the α=1 operating point (74.6/59.3 vs. 74.7/59.1, Table S10). We keep it off by default so that frame-level recall stays intact and the trade-off is left to the application. This gain cannot come from reweighting, because the weights are fixed once training ends.

In summary, each part of the Kairos framework is backed by direct evidence. As our main contribution, we reveal the duration bias of the traditional frame-level paradigm, support it with sufficient statistics (Fig. 1b–c, Eq. 1), and design Kairos accordingly, which provides the community with a better event-centric paradigm for training, evaluation and optimization. As a minor contribution, we verify its generalization ability through extensive experiments on four representative types of baselines (one-stage, two-stage, traditional debiasing and MLLM-based) under settings of different difficulty, PredCls and SGDet (Table 3). The event clock completes the framework as a decoupled component with its own measured benefits (Table 6, Table S10).

AuthorsOpenReview ↗

Response to Reviewer 7UC5 (Part 3/3)

Weakness 3: Annotation Flicker and Event Definition

Thank you for the valuable comment. We examined this question from three angles, and all of them indicate that the gains of Kairos come mainly from genuine short relations:

① The gains concentrate on genuine brief actions. The two predicates that gain most are eating (+10.3) and drinking from (+5.0, Table S6). For them, returning to the interrupted relation is the natural course of the action (e.g., holding, drinking, holding): 87% and 91% of their interior single-frame events do so, against 74% overall (L164–166). We further checked such cases against the video: every drinking event in the real clips of the new Fig. S2 (App. C) is a real sip, and the single frame of touching that Kairos recovers in Fig. 4 marks putting the object down or picking it up (L482–485).

② The gains follow duration and are not confined to single-frame events. Events of two to three frames, whose labels persist across consecutive keyframes, gain 3.0 points of ER@10 (56.9→59.9), close to the 3.5 points of single-frame events (42.8→46.3), while longer events stay in place (α=1, Fig. 3a). Across predicates, the gain falls with the mean duration (Spearman ρ=−0.73, Fig. 3b).

③ The conclusions hold under both event definitions used in the paper, and the analysis carries over to the suggested merge. The gains hold whether events are broken where the object disappears (App. B; ER1 +1.1 and ERtail +1.9 on average, Table 3) or merged across such gaps (App. D; ER1 +1.7 at unchanged R@10, Table 4). For the suggested merge, the identity of Eq. 1 holds for any definition of an event: the interrupted relation becomes one longer event, while the brief event that interrupts it remains a single-frame event, so the gap in duration behind the bias widens and the event-balanced loss still shifts weight to the brief event.

In summary, the three results consistently indicate that the gains of Kairos mainly reflect genuine brief relations. The real clips of Fig. S2 are added to App. C in the revised PDF. We hope this also answers your question on merging one-frame interruptions.

Questions 1–2: Illustration in Figure 1 and Concrete Examples

Thank you for both suggestions, which make the meaning of event time and ER@K clearer. Both are adopted in the revised PDF:

① Figure 1(a) now shows clock time and event time as two weighting schemes over the same frames and labels. The two bars count the same seven frames, and the width of each segment is its share of the weight: the drink carries 1/7 with one unit of time per frame and 1/3 with one unit per event. The outcome on the right is marked "trained with these weights": the drink is recovered because the model is trained with event weights, not because the evaluation is easier. For the same prediction, ER@K is in fact the stricter score: the clock-time transformer reaches 75.4 in R@10 but 56.9 in ER@10 (Table 4).

② Real examples that compare R@K and ER@K on the same predictions are added (Fig. S2, App. C). In three real Action Genome test clips, a person drinks from a cup between spells of holding it. The same prediction is scored on every clip: holding is recalled in every keyframe and drinking from is never reported. R@K stays at 76.9–86.2 (e.g., 10 of 13 labels), whereas ER@K is 33.3 on every clip, since the two missed sips are two of the three events. Similarly, on a real test track (Fig. 4a), the frame of touching that the clock-time transformer skips is one of 12 contact triplets but one of three events (ER 0.67, against 1.00 for Kairos; L482–485).

These examples show the practical meaning of ER@K: it asks whether each relation instance is found, however brief.

Thank you again for the insightful comments. If there are any remaining questions, we are more than happy to address them.

Reviewer_m2E2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies duration bias in Video Scene Graph Generation (VSGG), where conventional frame-level training and evaluation tend to favor long-lasting relations while overlooking short events. To address this issue, the authors propose Kairos, an event-centric framework that introduces event-aware evaluation metrics, duration-based loss reweighting, and a learned event clock for temporal modeling. Experiments on Action Genome show improved recall for short events and rare predicates while largely maintaining frame-level recall.

Critical strengths
  1. The paper identifies an important limitation of conventional frame-level evaluation in VSGG. The theoretical analysis of duration bias and the proposed event-aware metrics provide a useful perspective for evaluating temporal relations.

  2. The paper is well-written and logically organized. The controlled experiments show that event-balanced training improves the recall of short events and rare predicates without significantly sacrificing conventional recall.

Critical weaknesses
  1. The main performance gains appear to come from a relatively straightforward duration-based loss reweighting strategy, which builds on existing importance weighting techniques. Although the proposed event clock provides an interesting formulation, the ablation studies show that event-time attention offers little additional improvement. Therefore, the novelty of the overall method seems limited.

  2. All experiments are conducted on a single dataset, Action Genome, making it difficult to assess the generalizability of the findings. More importantly, the Kairos results across five backbones in Table 3 are marked as projected from the controlled study rather than directly measured. This weakens the empirical evidence for the claimed effectiveness and applicability of the framework.

Questions for authors
  1. Since event-time attention provides little improvement in Table 4, what additional benefit does the learned event clock offer compared with simply applying duration-based loss reweighting?

  2. Could the authors clarify how the projected improvements in Table 3 were obtained? Were the five host models actually trained and evaluated with Kairos, or were the improvements estimated from the controlled study? If the latter, can the authors provide direct experimental results to validate the claimed generalizability?

  3. How does the proposed event-centric approach compare with recent MLLM-based video scene graph generation methods, such as Cui et al. (2025) and Makarov et al. (2026)? Does duration bias also affect these models, and can the proposed method improve their performance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer m2E2 (Part 1/2)

We sincerely thank Reviewer m2E2 for the valuable comments. We are glad that the reviewer recognizes that our work identifies an important limitation of the conventional frame-level paradigm in VSGG and provides a useful perspective for evaluating temporal relations. We address each concern below, and all revisions are marked in red in the updated PDF.

Weakness 1 & Question 1: "The main performance gains appear to come from a relatively straightforward ... reweighting strategy ... the novelty of the overall method seems limited"

We agree that, compared with existing solutions that pursue increasingly elaborate temporal modeling, ours is simpler and more direct. This simplicity is by design, as it is directly guided by a precise diagnosis: our core contribution lies in identifying and defining the underlying problem. This study takes the first step towards event-centric VSGG, where relations are scored, learned and read out in event time rather than frame time (L97–102). Specifically,

① we reveal that the traditional frame-level paradigm, i.e., its training, its evaluation and the temporal modeling built on them, is dominated by long relations and systematically ignores short ones (L14–L17). In contrast, the event-centric view counts every relation instance once, however many frames it lasts, which places brief and long relations on an equal footing in both training and evaluation and offers a unified, principled basis for temporal relation reasoning.

② We support this finding with sufficient statistics: after three-frame smoothing of a perfect prediction, R@K retains 92.5% while single-frame events retain only 21.2% (Fig. 1b–c), and stronger temporal context improves events of four frames or more by 1.3 points but single-frame events by only 0.2 (Table 4). We describe this as the Duration Bias and quantify it exactly by β@K (Eq. 1, L91–L95).

③ We locate the source of this bias with a diagnostic experiment designed for this purpose: moving attention to event time, all else fixed, changes the overall metrics by at most half a point, whereas changing how training counts events raises ER1 by 1.7 at unchanged R@10 (Table 4). The bias therefore lies in what models count, not in how they pool (L122–124), which is why event-time attention brings little gain in the ablation.

④ We design Kairos accordingly: the remedy is dictated by the diagnosis rather than chosen empirically, i.e., weighting each event by d^(−α) (Eq. 8). Simple as it is, its gains fall where the diagnosis predicts (Spearman ρ=−0.73 between gain and predicate duration, Fig. 3b). Moreover, when combined with other methods, such simplicity typically simplifies the problem rather than adding difficulty: its gains hold across four types of base models (ER1 +1.1 and ERtail +1.9 on average, with R@20 within 0.2, Table 3). Beyond training, the event clock detects relation changes (AUROC 73.3 vs. 56.4 without supervision, Table 6) and reads out an already trained model without retraining (ER@10 +1.0 for 0.3 points of R@10, Table S10). These gains cannot come from reweighting, because the weights are fixed once training ends.

Defining the right problem so that its solution becomes simple is the core spirit of our work. We hope this response also addresses Question 1, i.e., why event-time attention brings little gain and what the event clock offers beyond reweighting.

AuthorsOpenReview ↗

Response to Reviewer m2E2 (Part 2/2)

Weakness 2 & Questions 2–3: Generalization ability of Kairos

We agree, and thank you for the suggestion to include MLLM-based methods, which is a strong complement to the verification of generalization. Since VSGG involves three settings (PredCls, SGCls and SGDet) and many candidate base models, the experiments required more than 1,500 H100 GPU hours. We cover four representative types of methods: one-stage, two-stage, traditional debiasing and MLLM-based. Each is trained and evaluated with its released code and the same evaluation, under two task settings that range from simple to complex, PredCls and SGDet. The results are given below and have been updated in the revised PDF (Table 3). Recall is the traditional frame-level metric; event recall (ER), single-frame event recall (ER1) and tail event recall (ERtail) are our event-level metrics (K=20; ↑/↓: change over the base model):

PredClsRecall (traditional)ER (ours)ER1 (ours)ERtail (ours)
One-stage76.155.640.416.5
+ Kairos76.0 (↓0.1)56.3 (↑0.7)41.5 (↑1.1)19.2 (↑2.7)
Two-stage71.852.438.88.8
+ Kairos71.8 (0.0)53.3 (↑0.9)40.2 (↑1.4)11.7 (↑2.9)
Debiased71.551.938.520.9
+ Kairos71.6 (↑0.1)52.9 (↑1.0)39.8 (↑1.3)23.0 (↑2.1)
MLLM-2 (Qwen2.5-VL-7B)73.553.340.120.2
+ Kairos73.6 (↑0.1)53.9 (↑0.6)41.5 (↑1.4)23.3 (↑3.1)
SGDetRecall (traditional)ER (ours)ER1 (ours)ERtail (ours)
One-stage41.024.715.76.5
+ Kairos41.0 (0.0)25.2 (↑0.5)16.5 (↑0.8)7.6 (↑1.1)
Two-stage34.119.712.64.2
+ Kairos34.0 (↓0.1)20.2 (↑0.5)13.3 (↑0.7)5.4 (↑1.2)
Debiased33.319.412.57.4
+ Kairos33.2 (↓0.1)19.8 (↑0.4)13.2 (↑0.7)8.3 (↑0.9)
MLLM-1 (Makarov et al., 2026)28.014.57.96.3
+ Kairos27.8 (↓0.2)15.0 (↑0.5)9.4 (↑1.5)7.5 (↑1.2)

Every result supports our core claim, which holds on each type of baseline: after adding Kairos, the traditional Recall stays unchanged (within 0.2 points), while all of our event-level metrics improve. On average, ER, ER1 and ERtail rise by 0.6, 1.1 and 1.9 points, the last being 17% of the tail's event recall, enabling these models to better serve downstream applications that are sensitive to short relations.

We are especially grateful for the suggestion on MLLM-based methods (Question 3). Kairos is complementary to them, so we insert it into them rather than compete with them. We include SceneGraphVLM (Makarov et al., 2026) as MLLM-1. For Cui et al. (2025), whose official code and checkpoints are not available, we follow its core design and use a LoRA-tuned Qwen2.5-VL-7B (Bai et al., 2025) as a suitable substitute (MLLM-2). The duration bias also affects these models: their event recall lies far below their frame-level recall (ER 14.5 vs. Recall 28.0 for MLLM-1; 53.3 vs. 73.5 for MLLM-2). Kairos improves both: ER1 rises from 7.9 to 9.4 and from 40.1 to 41.5, and ERtail from 6.3 to 7.5 and from 20.2 to 23.3 (Table 3d). Beyond our major contribution, i.e., an event-centric paradigm with better community evaluation standards, verifying the generalization of Kairos across diverse types of existing methods constitutes a minor contribution. Your suggestion greatly improves the completeness of our work.

Regarding the single dataset, Action Genome is the standard benchmark of the base models we build on, and we broaden the evidence within it along two axes: four types of base models and two task settings. The analysis itself does not depend on the dataset: the identity of Eq. 1 holds for any benchmark that scores relations of unequal length frame by frame (App. F).

We hope this response also addresses Questions 2 and 3. Thank you again for the insightful comments. If there are any remaining questions, we are more than happy to address them.

Dear Reviewer m2E2, thank you again for your constructive feedback and insightful suggestions, especially on the MLLM-based experiments, which greatly improve the completeness of our work and further verify the generalization ability of Kairos. If there are any remaining questions or suggestions, we would be more than happy to address them. Thank you for your time and effort in reviewing our paper!

Reviewer_SkB2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates duration bias in video scene graph generation, where frame-level training and evaluation give greater weight to long relations than brief events. It introduces event-level metrics and derives a covariance-based relationship between event recall and pooled frame-level recall. The proposed method, Kairos, combines inverse-duration loss weighting with supervised relation-change detection to construct an event clock. This clock can optionally adjust prediction logits and temporal attention using estimated relation persistence. Controlled experiments on Action Genome report improved recognition of short events and rare predicates while preserving frame-level recall.

Critical strengths
  • The paper highlights how frame-level training and evaluation can overlook brief but meaningful relations in video.
  • The covariance decomposition explicitly connects event recall to pooled frame-level recall, making duration bias interpretable and measurable. Duration-based loss weighting uses existing annotations and can be applied without changing the relation predictor’s architecture.
  • On Action Genome, the default configuration improves recall for short events and rare predicates while maintaining frame-level recall at the reported precision.
Critical weaknesses
  • Novelty is incremental, with a potentially useful formulation for VSGG. The ingredients have close precedents; the specific combination may be new, but the paper needs to establish what that combination contributes.
    • Duration imbalance: Already identified in temporal action segmentation, including the distinction between action frequency and action duration in G-TLA [1].
    • Event-time attention: Closely related to Forgetting Transformer’s cumulative gate bias [2].
    • Duration-based logit adjustment [3]
  • The paper states that single-frame events constitute 46.5% of events but only 11.3% of triplets, which makes the imbalance substantial. However, calling it a bias depends on the application: equal weighting of events is appropriate when recognizing each event matters; duration weighting remains reasonable for accuracy at a random moment. Downstream evidence would help justify that choice.
  • The event clock’s contribution is much less convincing. The default configuration disables both clock-based attention and readout, and the clock’s training gradient cannot affect the relation predictor. Under that configuration, the improvement comes from the inverse-duration loss weights. Appendix D explicitly confirms this. Event-time attention alone slightly reduces event recall in Table 4; clock-based readout offers additional event recall at a cost in frame recall.
  • The mathematical statement needs qualification. The covariance identity explains the sampling effect, but does not by itself establish that event weighting is universally preferable. I see the potential novelty mainly in the VSGG application and diagnosis, rather than the underlying importance-weighting mathematics.

[1] Pang, Zhanzhong, et al. "Long-tail temporal action segmentation with group-wise temporal logit adjustment." ECCV 2024.

[2] Lin, Zhixuan, et al. "Forgetting transformer: Softmax attention with a forget gate." ICLR 2025.

[3] Menon, Aditya Krishna, et al. "Long-tail learning via logit adjustment." ICLR 2021.

Questions for authors
  • What does the clock contribute beyond duration-weighted loss? The default configuration disables clock-based attention and readout, and stop-gradient prevents clock supervision from changing the relation predictor. Appendix D explicitly says the relation branch is the transformer trained with event-balanced loss. Which measured experiment demonstrates an additional benefit from the clock?

  • Why should one family clock represent the durations of different predicates? Suppose holding lasts 20 frames while drinking from overlaps for one frame. Both receive the same contact-family correction, −γlog⁡νtcon-\gamma \log \nu_t^{\mathrm{con}}, so their relative ranking remains unchanged. How does the readout correct this particular imbalance? Would predicate-specific clocks be necessary?

  • Can Eq. 10 be derived from the training objective? The loss assigns positive labels weights wtp=de(t,p)−α/Zw_t^p = d_{e(t,p)}^{-\alpha}/Z, whereas the readout applies s~tp=stp−γlog⁡νtg(p)\tilde{s}_t^p = s_t^p - \gamma \log \nu_t^{g(p)}. What assumptions connect these operations, particularly when predicate-level durations differ from family-level durations? Which target risk is optimized when both α>0\alpha > 0 and γ>0\gamma > 0, and when might they apply the duration correction twice?

  • What is new relative to an existing gate with the same supervision? With fu=e−κλuf_u = e^{-\kappa \lambda_u}, the causal attention bias can be written as bts=−κ(Λt−Λs)=∑u=s+1tlog⁡fub_{ts} = -\kappa(\Lambda_t-\Lambda_s) = \sum_{u=s+1}^{t}\log f_u. Can Kairos outperform a Forgetting Transformer supplied with the same boundary supervision and features? This comparison would separate the effect of supervision from the proposed parameterization.

  • What exactly does “event time” measure? Under the stated binary change model, the expected number of observed changes up to frame TT is ∑t=2T(1−e−λt)\sum_{t=2}^{T}(1-e^{-\lambda_t}). What additional assumption justifies interpreting ΛT=∑t=2Tλt\Lambda_T = \sum_{t=2}^{T}\lambda_t as an expected change count? How well calibrated are the persistence probabilities Sg(s,t)S^g(s,t) and duration estimates νtg\nu_t^g?

  • How sensitive is the method to the definition of an event? Appendix B breaks events when an object disappears, whereas Appendix D merges across those gaps. Does using one consistent definition change the gains? How robust are the results to annotation flicker and different frame-sampling rates, which directly change ded_e and therefore the weights de−αd_e^{-\alpha}?

  • Which experiments actually establish the claimed contribution? Table 3 labels the Kairos values as projections, while Section 5.2 describes measured results. Can the authors provide actual host-model runs, including comparisons against duration weighting alone and established imbalance corrections, evaluated with both standard frame-averaged recall and the proposed pooled metric?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SkB2 (Part 1/3)

We would like to sincerely thank Reviewer SkB2 for the constructive comments on our work.

We take every comment seriously and provide point-by-point responses below. All revisions are marked in red in the updated PDF. If there are any remaining questions, we are more than happy to address them.

Weakness 1: "... is a combination of imbalance [1], attention [2] and adjustment [3]...the paper needs to establish what that combination contributes"

Thank you for highlighting these relevant works. We acknowledge the contributions of [1–3], all of which are cited in our paper ([1] in Appendix L532, [2] in L147 and L329, [3] in L149, L312 and L479). [1, 3] offer effective corrections for the imbalance between classes, and [2] lets attention reweight the past. However, they all operate in frame time, where each frame counts once. Recall@K, the standard metric of VSGG, is itself computed in frame time: a relation is scored once for every frame it lasts. That is why mean Recall (mR@K), introduced to expose the long tail of predicates, does not remove this imbalance: it averages over classes, but inside each class it still counts frames. As a result, class-averaged recall remains 7.6 points above its event-level counterpart (mR@10 54.9 vs. mER@10 47.3, Table 4).

In contrast, we take the study of imbalance in VSGG one step further: we study the imbalance that the frame time paradigm itself creates, between brief and long instances of the same relation. The above three works form the basis of our research, whereas ours is not their combination.

To clarify, the operators that Kairos builds on, i.e., loss reweighting, logit adjustment [1, 3] and gated attention [2], are not the main focus and contribution of our work, and we did not claim them as our own; they have been widely studied for class imbalance and sequence modeling. Instead, this study takes the first step towards event-centric VSGG, where relations are scored, learned and read out in event time rather than frame time. We are inspired from a common failure mode in downstream applications (e.g., in real-time navigation that uses video scene graphs as the visual perception representation, VSGG models with higher Recall@20 [6, 7] keep improving on static relations (sitting on), but barely improve on instantaneous actions (touching, drinking from)) (this failure pattern is reported in Table 4). Through an in-depth analysis of these VSGG models,

① we reveal that the training, the evaluation, and almost follow-up research on temporal modeling in VSGG are all dominated by long relations, while short relations are systematically ignored (L14–L17).

② We support this finding with extensive statistics (in Fig. 1b–c, after three-frame smoothing of a perfect prediction, R@K still retains 92.5%, whereas single-frame events retain only 21.2%), and describe it as Duration Bias (L91–L95). Our core insight is that relations are essentially events of variable length, and pooled frame-level recall is exactly a length-biased average of event recall weighted by event duration, i.e., the inspection paradox of renewal theory. Therefore, the gap between the two can be written exactly in closed form as the covariance between event duration and event recall divided by the mean duration, which yields an interpretable and measurable diagnostic metric, β@K (Eq. 1). The frame-level training risk is the same length-biased average (Eq. 3), so this bias exists in both evaluation and learning. Based on this,

③ we design Kairos carefully, which accumulates an event clock for each predicate family, trains with an event-balanced loss that reweights each frame by its event duration, and at inference corrects predictions to event time through an event-time readout or attention bias, thereby concentrating the gains on short relations and tail predicates without modifying the base model's architecture and additional annotation. Kairos addresses the intrinsic limitation of current VSGG models, enabling them to better serve downstream applications such as navigation.

Given such fresh insights and remarkable merits, Kairos demonstrate sound motivation, solid solution, and commitment to better community evaluation standards, far beyond incremental innovation.

[6] Sayak Nag, Kyle Min, Subarna Tripathi, and Amit K. Roy-Chowdhury. Unbiased scene graph generation in videos. In CVPR, 2023.

[7] Guan Wang, Zhimin Li, Qingchao Chen, and Yang Liu. OED: Towards one-stage end-to-end dynamic scene graph generation. In CVPR, 2024.

AuthorsOpenReview ↗

Response to Reviewer SkB2 (Part 2/3)

Weakness 2: "Long events are not necessarily unreasonable..."

Thank you for this insightful comment. We fully agree that the appropriate weighting depends on the application: duration-weighted recall measures accuracy at any given moment, and event-weighted recall measures whether each relation instance is recognized. Accordingly, the Duration Bias does not imply that R@K is incorrect or that ER@K is universally superior. We use "bias" in the statistical sense of length-biased sampling, and Eq. 1 quantifies the gap between the two views exactly. Our tables therefore report R@K and ER@K side by side.

Precisely because the choice depends on the application, Kairos makes the trade-off explicit and controllable: α interpolates between frame-level (α=0) and event-level (α=1) learning, and γ moves a trained model along the R@K–ER@K trade-off without retraining (Fig. 3c, Appendix Table S10). At the default α=0.5, ER@10 improves by 1.1 points and single-frame recall by 1.7 points while R@10 stays at 75.4 (Table 4), so an application that only values frame-level accuracy loses nothing.

Event weighting matters when brief relations carry the semantics, as in navigation, video retrieval and question answering. Three existing results support this. First, three-frame smoothing retains 92.5% of R@K but only 21.2% of single-frame events (Fig. 1b–c). Second, the ten rarest predicates last 2.7 frames on average, against 4.8 for the five most frequent, and nine of them are contact actions (Appendix C). Third, recall at the onsets and offsets of relations rises from 53.3 to 54.4 (TR@10, Table 4). We agree that direct downstream evaluation would further strengthen this motivation, and we state its absence in the Limitations (Sec. 6).

Weakness 3 & Question 1: "The event clock's contribution is much less convincing ..."

We agree with the reviewer's reading of the results: in the default configuration the clock does not affect the predictions, event-time attention brings no gain, and the default gain comes from the event-balanced loss (App. D). The stop-gradient is a deliberate choice: trained jointly, the clock is as accurate, but the event recall of the tail predicates drops by 0.9 points on average (Table S8).

We see the absence of a gain from attention not as a missing contribution but as the experiment that locates the source of the duration bias. Table 4 changes one factor at a time: changing how temporal attention parameterizes time leaves event recall unchanged (ER1 −0.4), whereas changing how training counts events improves it (ER1 +1.7 at unchanged R@10). The duration bias therefore lies in what models count, not in how they pool. This mechanistic conclusion also explains a pattern in prior work: clock-time temporal context improves long events (+1.3 for events of four frames or more) but barely single-frame ones (+0.2), because it is learned under a frame-counting loss; better pooling alone cannot fix this.

Beyond this diagnostic role, the clock serves two functions that the loss cannot, and the first enables the second. First, relation-change detection: supervised by relation changes, the clock learns a signal with clear task semantics (AUROC 73.3 vs. 56.4 without supervision and 66.8 with a single shared clock, Table 6). Second, inference-time readout: since event durations are unknown at test time, the clock uses this signal, i.e., local persistence, to correct an already trained model without retraining, raising ER@10 by 1.0 for 0.3 points of R@10, whereas a class-level duration prior raises it by only about 0.2 for 0.5 (Fig. 3c, Table S10). We agree that this is a trade-off, and present it as an optional operating point (γ=0 recovers the default). The clock thus has a measured use independent of loss reweighting, which also answers Question 1. The revised PDF states this explicitly (Sec. 4.4): the loss is the main training-time correction, the clock its inference-time counterpart, and event-time attention an analysis that locates the bias.

AuthorsOpenReview ↗

Response to Reviewer SkB2 (Part 3/3)

Weakness 4: "The mathematical statement needs qualification ..."

We agree, and thank you for the suggestion. Right after Eq. 1, the revised PDF states: "The identity describes what frame-level recall measures; it does not by itself imply that event weighting is preferable, which depends on the application." We do not claim that event weighting is universally preferable. The sign of the bias is also empirical: β@K is positive when a model recovers long events better than short ones, which holds in every setting we report (Table 4, Table S4).

Thank you for acknowledging the novelty of our VSGG diagnosis. We systematically rethink the training and evaluation of VSGG models and reveal that the bias lies in what models count, and that Recall and mean Recall are ill-suited to downstream applications that rely on brief relations. Rather than the classical mathematics, which the paper attributes to its sources (Feller, 1971), our contribution is a direct way to measure and correct this bias: an exact and measurable quantity, β@K, and the event-balanced loss built on the same analysis.

Q2. Family vs. predicate clocks. Within one pair and frame, the family readout leaves the ranking among predicates of that family unchanged. That case is handled by the loss, whose weights use each predicate's own event duration (Eq. 8); the readout reorders triplets across pairs and families (Sec. 4.4). The remaining limitation is stated in App. D: a brief action that co-occurs with a longer relation of its family (drinking while holding) is almost never recalled under the with-constraint protocol. Finer clocks help change detection (AUROC 66.8 shared vs. 73.3 per family, Table 6); predicate-level clocks are left to future work.

Q3. Readout–loss consistency. Eq. 10 follows the logit-adjustment correspondence, not an exact derivation. For a calibrated sigmoid, weighting positives by w shifts the optimal logit by log w, so with ν ≈ d a model trained at α and read out at γ corresponds to weights d^−(α+γ): γ = 1−α targets the event risk, and α+γ > 1 over-corrects, i.e., applies the duration correction twice. Both assumptions are approximate: the α=0.5 model reaches the α=1 operating point at γ ≈ 1 rather than 0.5 (Table S10), so γ also acts as a calibration factor, chosen on the R@K–ER@K trade-off (Fig. 3c). The revised PDF states these assumptions (Sec. 4.4, L314–317).

Q4. Comparison with supervised forget gates. Given the same features and boundary supervision, the two coincide: our causal bias is exactly a forget gate supervised by relation changes (Table 2). This is the online setting of Table S9, where it changes R@10 by −0.2 and ER@10 by −0.3 relative to causal clock-time attention. Supervision changes what the gate detects (AUROC 73.3 vs. 56.4 without it, Table 6), not recall.

Q5. Meaning and calibration of event time. Λ is the cumulative hazard: it upper-bounds the expected number of changes and matches it only for small hazards, while S = exp(−ΔΛ) in Eq. 5 is exact. The revised PDF now reads "one unit of cumulative hazard, about one expected relation change when hazards are small" (Sec. 4.2, L270–271). We report discrimination (AUROC 73.3, Table 6) but have not measured the calibration of S or ν; the readout does not rely on it, since γ rescales log ν (Q3).

Q6. Robustness to event definition. Both definitions are already in use, and the gains appear under each: the controlled study merges runs across frames where the object is out of view (App. D), where ER1 rises by 1.7 (Table 4); Table 3 breaks runs at such frames (App. B), where ER1 rises by 1.1 on average. On flicker, the gains are not confined to interior single-frame events: recall at onsets and offsets rises by 1.1 (TR@10, Table 4), and the largest gains are on brief actions (eating +10.3, drinking from +5.0, Table S6). On sampling rate, a uniform change rescales all durations and cancels in the normalizer Z of Eq. 8, so only the rounding of short events matters; other rates are not evaluated.

Q7. Measured evidence for contributions. As stated in our response to Weakness 1, our major contribution, i.e., revealing the duration bias of the frame-level paradigm and designing Kairos accordingly, is supported by the statistics of Fig. 1b–c, Tables 4–5 and the analysis of Fig. 3, while our minor contribution on generalization ability is supported by Table 3. As suggested, we provide direct experimental results in the updated Table 3, including four representative types of base models (one-stage, two-stage, traditional debiasing and MLLM-based), each evaluated with both the traditional Recall and our event-level metrics. The "+ Kairos" rows use the event-balanced loss alone, and the traditional debiasing method serves as the established imbalance correction.

AuthorsOpenReview ↗

A gentle reminder before the discussion closes

Dear Reviewer SkB2, thank you again for your time and detailed review; we have thoroughly responded to the questions you raised. As a gentle reminder, the discussion period closes today. If there are any additional points you would like us to consider, please let us know. Your insights are invaluable to us, and we are eager to address any remaining issues to further improve the work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their valuable comments and suggestions!

We have updated the manuscript with the following revisions based on the reviewers' suggestions. All revisions in the updated version are highlighted in red:

  1. Add more experiments:
    • Add generalization ability experiments, covering four representative types of VSGG base models (one-stage, two-stage, traditional debiasing and MLLM-based) under the PredCls and SGDet settings, including two newly added MLLM-based models, SceneGraphVLM and a LoRA-tuned Qwen2.5-VL-7B (SkB2-Q7, m2E2-W2/Q2/Q3, 7UC5-W1; Table 3, L413–416).
  2. Add more discussion and details:
    • Add a clarification on the identity of Eq. 1 (SkB2-W4; L184–186).
    • Explain the role of each component in the default configuration: the event-balanced loss, the event clock and the readout (SkB2-W3/Q1, m2E2-Q1, 7UC5-W2; L365–366).
    • State the assumptions that connect the readout of Eq. 10 with the training loss (SkB2-Q3; L314–317).
    • Clarify the meaning of one unit of the event clock (SkB2-Q5; L270–271).
    • Revise Fig. 1(a) and its caption (7UC5-Q1; Fig. 1).
    • Add real examples that compare R@K and ER@K (7UC5-Q2; App. C, Fig. S2–S3).

Thanks again for all the effort and time, and we look forward to further discussions if there are any more questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.