- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Temporal Slot Activation (TSA) for object-centric video representation learning. TSA predicts a per-slot activation score and uses it to control both recurrent state updates and decoder participation. Inactive slots largely retain their previous states and receive reduced decoder attention. A temporal memory supports activation decisions, while usage and sparsity regularization encourage selective slot activation without visibility supervision. The method is evaluated on four video benchmarks, with substantial improvements on YouTube-VIS HQ and OVIS.
Critical strengths
- I think the paper addresses an important limitation of recurrent video OCL: a slot can lose its object identity when the object is temporarily invisible. Controlling both state updates and decoder participation is a sensible approach, and TSA adds little parameter overhead;
- The gains on YouTube-VIS HQ and OVIS are substantial across grouping and tracking metrics. For example, OVIS HOTA improves from 8.1 for the strongest reported baseline to 36.2. The main results are reported over three seeds, which makes these improvements more convincing;
- The comparisons include VideoSAUR, SlotContrast, RandSF.Q, and SmoothSA. The ablations examine both gating pathways, the activation regularizers, and temporal memory. I also appreciate the occlusion-duration analysis and the additional representation and downstream evaluations in the appendix.
Critical weaknesses
- Table 3 and Appendix G.2 are useful, but TSA also improves substantially when ( ). Moreover, a smaller pre/post-occlusion slot distance does not by itself show that the original slot has recovered the same object. My main concern is how much of the improvement comes specifically from preserving identity through absence. Re-association accuracy and identity-switch rates around occlusion events, evaluated on correctly segmented visible objects, together with the DetA/AssA components of HOTA, would make this claim more convincing;
- Tables 4 and 9 show that gating both pathways is better than gating either one alone. However, this does not establish the benefit of using the same score for both decisions, which is a central design claim. Under partial occlusion, the confidence needed to update a slot may differ from that needed to decode its visible region. I would appreciate a comparison with independently learned update and decoder gates under comparable training objectives and budgets;
- Eq. 7 uses the absolute activation score to control state updates, whereas Eq. 8 normalizes scores across slots. For fixed attention logits, multiplying all scores by the same positive factor leaves decoder attention unchanged. Thus, a small activation does not necessarily imply negligible decoder participation unless other slots remain sufficiently active. I think the paper should state this condition and explain how training keeps visible slots active. This matters for the claim that one score consistently controls inactivity in both pathways;
- Section 3 offers a useful motivation, but Eq. 3 is an upper bound on accumulated state change; it does not establish that updates align or that drift necessarily grows with absence duration. Eq. 4 identifies a possible reconstruction-interference pathway, but a nonzero gradient alone does not establish identity corruption. I think these are plausible failure mechanisms, but the assumptions should be explicit and the conclusions framed as mechanisms supported by the experiments, rather than necessary consequences of unconditional slot propagation.
Questions for authors
- Can you report re-association accuracy and identity switches around occlusion events, while controlling for visible-frame segmentation quality?
- How does TSA compare with a model using separate update and decoder activation scores?
- What prevents all activation scores from becoming small in the same frame, and under what conditions does Eq. 8 suppress an inactive slot?
- Which conclusions in Section 3 follow directly from the equations, and which rely on assumptions about update directions or reconstruction gradients?
4 comments
Response to Reviewer qoiE (Part 2/2)
Q3 / W3. Scale invariance of Eq. 8 and what keeps visible slots active
We thank the reviewer for this observation. The training objective discourages the case where all scores become low in the same frame. If all scores were low, Eq. 7 would prevent every slot from taking in current-frame evidence, so the slots could no longer explain the current frame and the reconstruction error would increase. The gradient of the reconstruction loss through Eq. 7 forces the model to raise the scores of the slots that explain visible content, counterbalancing , while pushes each score toward 0 or 1 and discourages ambiguous values. Eq. 8 suppresses a slot when its score is much lower than the scores of the other slots. Since slots of visible objects stay close to 1, this holds for a slot with a low score, whose state is protected, so it also does not take part in decoding. The single score therefore controls both pathways consistently.
Q4 / W4. Which conclusions in Sec. 3 follow from the equations
We thank the reviewer for this question. In Sec. 3, the equations provide the motivation for TSA and show that both failure pathways can occur under unconditional propagation. We then confirm through experiments that they actually harm identity in practice.
Pathway I (state drift). Eq. 3 is an upper bound, and the statement that drift grows with absence duration relies on our assumption that successive updates do not cancel out. Our original wording could suggest that this follows from the equation itself. This assumption is supported experimentally: under unconditional propagation, the drift of a slot across an occlusion generally grows with the occlusion length on all four datasets (App. G.2). In the revised paper, Sec. 3 now links it to this evidence.
Pathway II (reconstruction interference). Eq. 4 shows that the slot of an absent object still receives gradients from the reconstruction loss. The statement that this harms its identity relies on an assumption, namely that these gradients pull the slot toward unrelated visible content. Our experiments support this. In Table. 4 (left), variants #3 and #4 both gate the state update and differ only in whether this pathway is also blocked in the decoder. Blocking it raises HOTA from 21.2 to 39.4, which shows that the pathway harms identity in practice. In the revised paper, Sec. 3 now links this pathway to this evidence.
We have also revised our contribution in Sec. 1 to state that this analysis is supported empirically.
Thank you for the additional experiments. The new results address most of my concerns, particularly the evidence for identity recovery after occlusion.
The IDSW analysis directly tests whether a reappearing object returns to the same slot, which was missing from the original representation-drift analysis. The reduction in identity switches is substantial, and the DetA/AssA breakdown gives a clearer picture of the improvements in both segmentation and association. The comparison with separate update and decoder gates also answers my question about the shared-score design. I find the clarification of the assumptions in Sec. 3 reasonable and am comfortable with the analysis serving as motivation supported by the experiments.
I have one remaining clarification about Tab. R1. Are all methods evaluated on a common subset of occlusion events where they successfully segment the object before and after the gap, or does each method select its own qualifying events? If the subsets differ, the methods may be evaluated on events of different difficulty. Please clarify this and report the number of retained events for each method.
Overall, the rebuttal has addressed my main reservations. I maintain my positive recommendation with greater confidence, and would like to see the new experiments and evaluation details included in the revised paper.
Clarification on Retained Events in Table R1
Thank you for the feedback. We are glad that the new experiments addressed your concerns, and we clarify Table R1 below.
First, all methods have been evaluated on the same split trainset/testset.
Depend on model design in obtaining slot, each method is evaluated on its own retained events. Following the suggested protocol, we keep only the occlusion events in which the object is correctly segmented before and after its absence, matched with IoU ≥ 0.5, for each method. Since each method segments different objects correctly, the retained events differ across methods. This is why we report the IDSW rate, which reflects how often identity switches occur relative to each method's own retained events, rather than the raw count. On average over 3 seeds, RandSF.Q, SmoothSA, and TSA retain 84.0, 74.0, and 30.3 occlusion events, respectively.
To compare the methods on events of the same difficulty, we also evaluate all three methods on the events retained by all of them. We report this at IoU ≥ 0.5 and at a more lenient threshold, IoU ≥ 0.2, which more than doubles the number of shared events:
Table R4. IDSW rate (%) on the occlusion events retained by all methods on OVIS.
| IoU | Shared retained events | RandSF.Q | SmoothSA | TSA |
|---|---|---|---|---|
| 0.5 | 24.3 ± 0.9 | 75.1 ± 7.7 | 76.8 ± 3.4 | 30.1 ± 4.9 |
| 0.2 | 57.0 ± 2.9 | 69.9 ± 6.4 | 75.6 ± 6.0 | 34.6 ± 6.2 |
In both settings, TSA switches identity in about 30–35% of the events, compared with about 70–77% for the baselines. This is consistent with Table R1, so the advantage of TSA is not due to differences in the retained events.
Following the reviewer's suggestion, we have added the IDSW analysis and the ablation with separate update and decoder scores to the revised paper.
Response to Reviewer qoiE (Part 1/2)
We thank the reviewer for the careful review of our work and for the helpful questions, which helped us improve the paper. We address each point below.
Q1 / W1. How much of the improvement comes from preserving identity through absence?
App. G.2 is designed to show that the gated update preserves the slot state across an absence, which is the prerequisite for recovering the same object afterwards. We agree with the reviewer that this does not by itself show the recovery, and we thank the reviewer for suggesting a more direct measurement. We therefore report the two analyses the reviewer requested.
Re-association around occlusion events. To control for segmentation quality, we keep only the occlusion events in which the object is correctly segmented both before and after its absence, i.e. matched in its last visible frame before the gap and in its first visible frame after it. For each event, we check whether the first re-match after the gap is an identity switch. Results on OVIS over all occlusion events:
Table. R1. Identity switch (IDSW) rate around occlusion events on OVIS.
DetA / AssA. We decompose HOTA into detection (DetA) and association (AssA) accuracy on the full evaluation set:
Table. R2. HOTA decomposed into detection (DetA) and association (AssA) accuracy on OVIS and YouTube-VIS HQ.
When an object is correctly segmented before and after an occlusion, the baselines switch identity in most of these events (80.5% for RandSF.Q and 76.5% for SmoothSA), while TSA does so in only 31.8%. TSA thus binds the reappearing object back to the same slot far more often than the baselines. TSA also substantially improves both DetA and AssA on both datasets, reflecting better segmentation and identities that are kept consistent over time. Together, these results show that a large part of the improvement comes from preserving identity through absence.
Q2 / W2. Shared vs. separate activation scores
We thank the reviewer for suggesting this comparison, which directly tests a central design choice of TSA. Tables 4 and 9 show that gating both pathways is better than gating either one. Below, we first explain why the two pathways should share one score, and then compare TSA with independently learned gates.
Why one score. The two decisions answer the same question: is the object of this slot present in the current frame? If it is, the slot should both take in current-frame evidence and explain that evidence in decoding. If it is not, it should do neither. Since both pathways are controlled by the same score, they always move together: a slot whose state is held is also down-weighted in decoding, and a slot that is updated also takes part in decoding. With independent scores, the two decisions can disagree, which can leave a slot in an inconsistent state:
A shared score rules out both states by construction. It also receives training signal from both pathways (Eq. 7 and Eq. 8). Separate scores each learn from only one pathway, so nothing keeps the two decisions consistent, and each score has less signal to learn from.
Comparison with separate scores. We compare TSA with a variant that learns two independent scores: one for the state update (Eq. 7) and the other for decoder participation (Eq. 8). Each score is predicted by its own estimator and regularized with its own Lreg. All other settings, including training schedule and evaluation, are identical (YouTube-VIS HQ):
Table. R3. Shared vs. separate activation scores on YouTube-VIS HQ.
The shared score is better on all metrics, with the largest gains on the tracking metrics (+4.5 HOTA, +5.6 IDF1). This supports the design argument in Sec. 3: keeping the two pathways consistent matters more than allowing them to differ.