acceptodds
Under review as a conference paper at ICLR 2027

TSA: Temporal Slot Activation for Persistent Object-Centric Video Representation

Abstract

Unsupervised video object-centric learning aims to decompose dynamic scenes into temporally persistent entity representations. Existing recurrent video slot-attention methods propagate a fixed set of slots across frames, but typically assume *unconditional slot propagation*: every slot is updated and decoded at every frame, regardless of whether its corresponding object is visible. We show that this design violates a basic lifecycle requirement for persistent slots: when an object is absent or fully occluded, its slot should preserve its previous state and avoid explaining unrelated visible content. Instead, unconditional propagation creates *update-induced state drift*, where current-frame evidence overwrites the absent object's representation, and *decoder-induced reconstruction interference*, where the inactive slot remains coupled to reconstruction through decoder attention. We propose **Temporal Slot Activation (TSA)**, a lightweight mechanism that learns a per-slot, per-frame activation score without visibility supervision. TSA uses this activation as a shared latent control variable for slot lifecycle modeling. When a slot is inactive, TSA anchors its state to the previous slot through activation-gated updating and suppresses its decoder participation through an activation-dependent additive bias on attention logits before softmax normalization. This jointly reduces state drift and reconstruction-driven interference. To improve decisions under partial occlusion and gradual reappearance, TSA further conditions activation prediction on a per-slot temporal memory produced by a Temporal Context Encoder. We evaluate TSA on MOVi-C, MOVi-E, YouTube-VIS HQ, and the occlusion-heavy OVIS benchmarks, using both standard metrics (FG-ARI, mBO) and tracking-based metrics (IDF1, HOTA). TSA improves object decomposition and temporal identity preservation, with large gains on long, heavily occluded videos. Source code will be released upon publication.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
video object-centric learning, slot-based representation learning, temporal consistency, slot lifecycle

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qoiEOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal Slot Activation (TSA) for object-centric video representation learning. TSA predicts a per-slot activation score and uses it to control both recurrent state updates and decoder participation. Inactive slots largely retain their previous states and receive reduced decoder attention. A temporal memory supports activation decisions, while usage and sparsity regularization encourage selective slot activation without visibility supervision. The method is evaluated on four video benchmarks, with substantial improvements on YouTube-VIS HQ and OVIS.

Critical strengths
  1. I think the paper addresses an important limitation of recurrent video OCL: a slot can lose its object identity when the object is temporarily invisible. Controlling both state updates and decoder participation is a sensible approach, and TSA adds little parameter overhead;
  2. The gains on YouTube-VIS HQ and OVIS are substantial across grouping and tracking metrics. For example, OVIS HOTA improves from 8.1 for the strongest reported baseline to 36.2. The main results are reported over three seeds, which makes these improvements more convincing;
  3. The comparisons include VideoSAUR, SlotContrast, RandSF.Q, and SmoothSA. The ablations examine both gating pathways, the activation regularizers, and temporal memory. I also appreciate the occlusion-duration analysis and the additional representation and downstream evaluations in the appendix.
Critical weaknesses
  1. Table 3 and Appendix G.2 are useful, but TSA also improves substantially when ( TΔ=0T_\Delta=0 ). Moreover, a smaller pre/post-occlusion slot distance does not by itself show that the original slot has recovered the same object. My main concern is how much of the improvement comes specifically from preserving identity through absence. Re-association accuracy and identity-switch rates around occlusion events, evaluated on correctly segmented visible objects, together with the DetA/AssA components of HOTA, would make this claim more convincing;
  2. Tables 4 and 9 show that gating both pathways is better than gating either one alone. However, this does not establish the benefit of using the same score for both decisions, which is a central design claim. Under partial occlusion, the confidence needed to update a slot may differ from that needed to decode its visible region. I would appreciate a comparison with independently learned update and decoder gates under comparable training objectives and budgets;
  3. Eq. 7 uses the absolute activation score to control state updates, whereas Eq. 8 normalizes scores across slots. For fixed attention logits, multiplying all scores by the same positive factor leaves decoder attention unchanged. Thus, a small activation does not necessarily imply negligible decoder participation unless other slots remain sufficiently active. I think the paper should state this condition and explain how training keeps visible slots active. This matters for the claim that one score consistently controls inactivity in both pathways;
  4. Section 3 offers a useful motivation, but Eq. 3 is an upper bound on accumulated state change; it does not establish that updates align or that drift necessarily grows with absence duration. Eq. 4 identifies a possible reconstruction-interference pathway, but a nonzero gradient alone does not establish identity corruption. I think these are plausible failure mechanisms, but the assumptions should be explicit and the conclusions framed as mechanisms supported by the experiments, rather than necessary consequences of unconditional slot propagation.
Questions for authors
  1. Can you report re-association accuracy and identity switches around occlusion events, while controlling for visible-frame segmentation quality?
  2. How does TSA compare with a model using separate update and decoder activation scores?
  3. What prevents all activation scores from becoming small in the same frame, and under what conditions does Eq. 8 suppress an inactive slot?
  4. Which conclusions in Section 3 follow directly from the equations, and which rely on assumptions about update directions or reconstruction gradients?
4 comments
AuthorsOpenReview ↗

Response to Reviewer qoiE (Part 1/2)

We thank the reviewer for the careful review of our work and for the helpful questions, which helped us improve the paper. We address each point below.

Q1 / W1. How much of the improvement comes from preserving identity through absence?

App. G.2 is designed to show that the gated update preserves the slot state across an absence, which is the prerequisite for recovering the same object afterwards. We agree with the reviewer that this does not by itself show the recovery, and we thank the reviewer for suggesting a more direct measurement. We therefore report the two analyses the reviewer requested.

Re-association around occlusion events. To control for segmentation quality, we keep only the occlusion events in which the object is correctly segmented both before and after its absence, i.e. matched in its last visible frame before the gap and in its first visible frame after it. For each event, we check whether the first re-match after the gap is an identity switch. Results on OVIS over all occlusion events:

Table. R1. Identity switch (IDSW) rate around occlusion events on OVIS.

IDSW rate (%) ↓\downarrow
RandSF.Q80.5 ± 0.9
SmoothSA76.5 ± 3.4
TSA31.8 ± 9.3

DetA / AssA. We decompose HOTA into detection (DetA) and association (AssA) accuracy on the full evaluation set:

Table. R2. HOTA decomposed into detection (DetA) and association (AssA) accuracy on OVIS and YouTube-VIS HQ.

OVIS DetAOVIS AssAYT-VIS HQ DetAYT-VIS HQ AssA
RandSF.Q4.7 ± 0.113.5 ± 0.613.3 ± 0.237.2 ± 0.3
SmoothSA5.0 ± 0.014.7 ± 0.913.5 ± 0.340.6 ± 0.7
TSA27.2 ± 1.356.2 ± 2.531.5 ± 5.166.0 ± 2.7

When an object is correctly segmented before and after an occlusion, the baselines switch identity in most of these events (80.5% for RandSF.Q and 76.5% for SmoothSA), while TSA does so in only 31.8%. TSA thus binds the reappearing object back to the same slot far more often than the baselines. TSA also substantially improves both DetA and AssA on both datasets, reflecting better segmentation and identities that are kept consistent over time. Together, these results show that a large part of the improvement comes from preserving identity through absence.

Q2 / W2. Shared vs. separate activation scores

We thank the reviewer for suggesting this comparison, which directly tests a central design choice of TSA. Tables 4 and 9 show that gating both pathways is better than gating either one. Below, we first explain why the two pathways should share one score, and then compare TSA with independently learned gates.

Why one score. The two decisions answer the same question: is the object of this slot present in the current frame? If it is, the slot should both take in current-frame evidence and explain that evidence in decoding. If it is not, it should do neither. Since both pathways are controlled by the same score, they always move together: a slot whose state is held is also down-weighted in decoding, and a slot that is updated also takes part in decoding. With independent scores, the two decisions can disagree, which can leave a slot in an inconsistent state:

  • Updated but not decoded. The slot keeps absorbing current-frame features, yet it is not tied to any region through reconstruction. Its state can therefore drift without any correction.
  • Held but decoded. The slot's state is protected, yet it still competes in decoding and receives reconstruction gradients from unrelated visible content (Pathway II).

A shared score rules out both states by construction. It also receives training signal from both pathways (Eq. 7 and Eq. 8). Separate scores each learn from only one pathway, so nothing keeps the two decisions consistent, and each score has less signal to learn from.

Comparison with separate scores. We compare TSA with a variant that learns two independent scores: one for the state update (Eq. 7) and the other for decoder participation (Eq. 8). Each score is predicted by its own estimator and regularized with its own LregL_{\text{reg}}. All other settings, including training schedule and evaluation, are identical (YouTube-VIS HQ):

Table. R3. Shared vs. separate activation scores on YouTube-VIS HQ.

FG-ARImBOHOTAIDF1
Separate scores73.754.141.843.6
Shared score (TSA)76.355.846.349.2

The shared score is better on all metrics, with the largest gains on the tracking metrics (+4.5 HOTA, +5.6 IDF1). This supports the design argument in Sec. 3: keeping the two pathways consistent matters more than allowing them to differ.

AuthorsOpenReview ↗

Response to Reviewer qoiE (Part 2/2)

Q3 / W3. Scale invariance of Eq. 8 and what keeps visible slots active

We thank the reviewer for this observation. The training objective discourages the case where all scores become low in the same frame. If all scores were low, Eq. 7 would prevent every slot from taking in current-frame evidence, so the slots could no longer explain the current frame and the reconstruction error would increase. The gradient of the reconstruction loss through Eq. 7 forces the model to raise the scores of the slots that explain visible content, counterbalancing LusageL_{\text{usage}}, while LsparseL_{\text{sparse}} pushes each score toward 0 or 1 and discourages ambiguous values. Eq. 8 suppresses a slot when its score is much lower than the scores of the other slots. Since slots of visible objects stay close to 1, this holds for a slot with a low score, whose state is protected, so it also does not take part in decoding. The single score therefore controls both pathways consistently.

Q4 / W4. Which conclusions in Sec. 3 follow from the equations

We thank the reviewer for this question. In Sec. 3, the equations provide the motivation for TSA and show that both failure pathways can occur under unconditional propagation. We then confirm through experiments that they actually harm identity in practice.

Pathway I (state drift). Eq. 3 is an upper bound, and the statement that drift grows with absence duration relies on our assumption that successive updates do not cancel out. Our original wording could suggest that this follows from the equation itself. This assumption is supported experimentally: under unconditional propagation, the drift of a slot across an occlusion generally grows with the occlusion length on all four datasets (App. G.2). In the revised paper, Sec. 3 now links it to this evidence.

Pathway II (reconstruction interference). Eq. 4 shows that the slot of an absent object still receives gradients from the reconstruction loss. The statement that this harms its identity relies on an assumption, namely that these gradients pull the slot toward unrelated visible content. Our experiments support this. In Table. 4 (left), variants #3 and #4 both gate the state update and differ only in whether this pathway is also blocked in the decoder. Blocking it raises HOTA from 21.2 to 39.4, which shows that the pathway harms identity in practice. In the revised paper, Sec. 3 now links this pathway to this evidence.

We have also revised our contribution in Sec. 1 to state that this analysis is supported empirically.

Thank you for the additional experiments. The new results address most of my concerns, particularly the evidence for identity recovery after occlusion.

The IDSW analysis directly tests whether a reappearing object returns to the same slot, which was missing from the original representation-drift analysis. The reduction in identity switches is substantial, and the DetA/AssA breakdown gives a clearer picture of the improvements in both segmentation and association. The comparison with separate update and decoder gates also answers my question about the shared-score design. I find the clarification of the assumptions in Sec. 3 reasonable and am comfortable with the analysis serving as motivation supported by the experiments.

I have one remaining clarification about Tab. R1. Are all methods evaluated on a common subset of occlusion events where they successfully segment the object before and after the gap, or does each method select its own qualifying events? If the subsets differ, the methods may be evaluated on events of different difficulty. Please clarify this and report the number of retained events for each method.

Overall, the rebuttal has addressed my main reservations. I maintain my positive recommendation with greater confidence, and would like to see the new experiments and evaluation details included in the revised paper.

AuthorsOpenReview ↗

Clarification on Retained Events in Table R1

Thank you for the feedback. We are glad that the new experiments addressed your concerns, and we clarify Table R1 below.

First, all methods have been evaluated on the same split trainset/testset.

Depend on model design in obtaining slot, each method is evaluated on its own retained events. Following the suggested protocol, we keep only the occlusion events in which the object is correctly segmented before and after its absence, matched with IoU ≥ 0.5, for each method. Since each method segments different objects correctly, the retained events differ across methods. This is why we report the IDSW rate, which reflects how often identity switches occur relative to each method's own retained events, rather than the raw count. On average over 3 seeds, RandSF.Q, SmoothSA, and TSA retain 84.0, 74.0, and 30.3 occlusion events, respectively.

To compare the methods on events of the same difficulty, we also evaluate all three methods on the events retained by all of them. We report this at IoU ≥ 0.5 and at a more lenient threshold, IoU ≥ 0.2, which more than doubles the number of shared events:

Table R4. IDSW rate (%) on the occlusion events retained by all methods on OVIS.

IoUShared retained eventsRandSF.QSmoothSATSA
0.524.3 ± 0.975.1 ± 7.776.8 ± 3.430.1 ± 4.9
0.257.0 ± 2.969.9 ± 6.475.6 ± 6.034.6 ± 6.2

In both settings, TSA switches identity in about 30–35% of the events, compared with about 70–77% for the baselines. This is consistent with Table R1, so the advantage of TSA is not due to differences in the retained events.

Following the reviewer's suggestion, we have added the IDSW analysis and the ablation with separate update and decoder scores to the revised paper.

Reviewer_GuiXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Temporal Slot Activation (TSA) to improve temporal identity preservation in unsupervised video object-centric learning. The authors identify unconditional slot propagation as a key limitation of existing video slot attention methods, causing representation drift and identity switches when objects become occluded or disappear. To address this issue, TSA introduces a learned per-slot activation score that jointly controls slot state updates and decoder participation, while leveraging temporal memory for more reliable activation decisions. Experiments on synthetic and real-world benchmarks demonstrate improvements in object decomposition and temporal consistency, particularly in challenging scenarios involving severe occlusion.

Critical strengths
  1. The paper identifies an important limitation of existing video slot attention methods and provides a clear analysis of how unconditional slot propagation can cause representation drift and identity switches.

  2. The proposed activation gating mechanism is conceptually simple and achieves substantial improvements on real-world benchmarks, particularly OVIS, demonstrating its potential for preserving object identities under severe occlusion.

Critical weaknesses

Limited robustness to dataset-specific hyperparameters

  • The method relies on substantially different regularization hyperparameters across datasets, raising concerns about its robustness and dependence on dataset-specific tuning. In particular, the regularization weights vary considerably across benchmarks, suggesting that the method may require careful tuning for each dataset. In addition, additional sensitivity analyses across multiple datasets, ideally using shared hyperparameter settings, are needed to establish the robustness and generalizability of the proposed method.

Lack of sensitivity analysis on the number of slots

  • Since the proposed activation gating mechanism is designed to deactivate unnecessary slots, it should ideally maintain relatively stable performance as the number of slots increases beyond the actual number of objects. However, the paper does not investigate how performance changes with different slot budgets. An ablation study varying the number of slots is necessary to verify whether the proposed gating mechanism effectively handles redundant slots and remains robust to over-provisioned slot budgets.

Incomplete comparisons with recent related methods

  • Several recent methods discussed in the Related Work section are not included in the main quantitative comparisons. In particular, SlotCurri addresses a closely related problem of redundant slot representations, making it a relevant baseline.
Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer GuiX (Part 1/2)

We thank the reviewer for recognizing the importance of the problem we address and the effectiveness of TSA, and for the valuable suggestions, which led to additional analyses that further support our findings. We address each concern below.

W1. Robustness to dataset-specific hyperparameters

We thank the reviewer for raising this point. The regularization weights indeed differ across datasets. Each dataset has different characteristics, for example how many objects are visible per frame and how often they are absent, and in unsupervised object-centric learning, where no labels define what an object is, hyperparameters such as the slot budget and loss weights shape the decomposition the model learns and are commonly set per dataset. However, TSA does not depend on these values to outperform the baselines. On YouTube-VIS HQ, even the worst setting in the whole sweep of Table. 8 stays above the best baseline on every metric (e.g. 33.1 vs. 23.8 HOTA and 30.5 vs. 19.3 IDF1). As requested, we further report (i) a sensitivity analysis on a second dataset and (ii) results with a single setting shared across all four benchmarks.

(i) Sensitivity on a second dataset. We repeat the analysis of App. E on MOVi-C, varying one hyperparameter over a 3 ×\times range while keeping the other at its paper value:

Table. R1. Sensitivity to λreg\lambda_{\text{reg}} on MOVi-C (β\beta = 1). The paper setting is in bold.

λreg\lambda_{\text{reg}} (β\beta = 1)FG-ARImBOHOTAIDF1
0.0374.429.233.732.1
0.04575.529.734.432.4
0.0675.430.135.233.4
0.07575.530.435.533.6
0.0974.330.235.433.3

Table. R2. Sensitivity to β\beta on MOVi-C (λreg\lambda_{\text{reg}} = 0.06). The paper setting is in bold.

β\beta (λreg\lambda_{\text{reg}}= 0.06)FG-ARImBOHOTAIDF1
0.575.730.034.933.2
0.7575.030.135.033.4
1.075.430.135.233.4
1.2576.330.335.333.2
1.575.129.634.632.7

All metrics stay within 2 points across both sweeps, and FG-ARI remains above the best baseline (69.0) in every setting.

(ii) A single shared setting. We apply the MOVi-C setting (λreg\lambda_{\text{reg}} = 0.06, β\beta = 1) to all four datasets:

Table. R3. Per-dataset paper setting vs. a single shared setting (λreg\lambda_{\text{reg}} = 0.06, β\beta = 1) on all four datasets.

DatasetSettingFG-ARImBOHOTAIDF1
MOVi-Cpaper = shared75.430.135.233.4
MOVi-Epaper83.424.627.115.6
shared81.725.228.017.1
YT-VIS HQpaper76.355.846.349.2
shared72.350.136.236.5
OVISpaper61.134.439.438.9
shared45.625.214.19.7

On the synthetic datasets, the shared setting performs on par with the paper setting, and on MOVi-E it remains as competitive with the baselines as the paper setting. On YouTube-VIS HQ and OVIS, it still outperforms the best baseline on every metric.

In summary, the hyperparameters affect how much TSA improves, but TSA does not need per-dataset values to outperform the baselines; per-dataset values only add further improvement.

We have added both results to App. E of the revised paper (new Tables 10 and 11).

AuthorsOpenReview ↗

Response to Reviewer GuiX (Part 2/2)

W2. Sensitivity to the number of slots

We thank the reviewer for suggesting this ablation. Before presenting the results, we briefly clarify the role of the activation gate in TSA.

TSA does not target the number of slots, but how each slot evolves over time. At every frame, the activation score decides whether a slot is updated from the current frame and whether it takes part in decoding. Its purpose is temporal consistency: the slot of a temporarily absent object is protected, so that it can bind back to the same object when the object reappears. Fewer slots taking part in a frame is a side effect of this mechanism, not its goal.

This differs from adaptive slot methods such as AdaSlot[1] and MetaSlot[2]. These methods remove the slots judged redundant from the current frame only. At the next frame, the selection is made again from the new content, so a slot is not tied to the object it represented before and may bind to any object. In TSA, no slot is ever removed. A protected slot keeps its state and returns as soon as its object reappears. Adapting the slot budget to the scene is the goal of that separate line of work and lies outside the scope of TSA.

Nevertheless, as requested by the reviewer, we vary the slot budget on YouTube-VIS HQ (default KK = 7), keeping all other hyperparameters fixed:

Table. R4. Effect of the slot budget KK on YouTube-VIS HQ.

KKFG-ARImBOHOTAIDF1
776.355.846.349.2
876.252.538.238.8
974.852.337.538.0

Increasing the slot budget mainly affects the tracking metrics, which decrease at larger budgets, while segmentation quality remains stable (FG-ARI within 1.5 points). Across the tested budgets, TSA remains above the best baseline at its default budget.

References

[1] Fan, K., Bai, Z., Xiao, T., He, T., Horn, M., Fu, Y., ... & Zhang, Z. (2024, June). Adaptive slot attention: Object discovery with dynamic slot number. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 23062-23071). IEEE.

[2] Liu, H., Zhao, R., Chen, H., & Pajarinen, J. (2025). MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning. In 39th Conference on Neural Information Processing Systems, NeurIPS 2025.

W3. Comparison with recent related methods and SlotCurri

We appreciate this suggestion and clarify our choice of baselines and the relation between SlotCurri and TSA below.

Choice of baselines. TSA addresses how slots are carried over time in recurrent video slot attention. We therefore compare it with methods from the same line: VideoSAUR, SlotContrast, RandSF.Q and SmoothSA. RandSF.Q and SmoothSA are among the most recent methods in this line. These methods also share the same evaluation protocol, which allows a fair comparison under identical conditions.

Why SlotCurri was not included. SlotCurri addresses a different problem. TSA decides, at every frame, how each slot is updated and how much it takes part in decoding. SlotCurri targets over-fragmentation, where one object is split across several slots: training starts with a few slots and progressively adds new ones where reconstruction error remains high. At inference, all KK slots are still updated and decoded at every frame, so slots of absent objects are handled as in unconditional propagation. SlotCurri also follows a different protocol, so its reported numbers are not directly comparable to ours.

Nevertheless, for completeness, we reproduce SlotCurri under the same protocol on YouTube-VIS HQ, following its official hyperparameters, and compare it with TSA:

Table. R5. Comparison with SlotCurri on YouTube-VIS HQ under our protocol.

MethodFG-ARImBOHOTAIDF1
SlotCurri42.824.919.818.0
TSA76.355.846.349.2

TSA outperforms SlotCurri by a large margin on all metrics, in both segmentation and tracking.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer GuiX,

Thank you again for your review. We have addressed your concerns.

We are waiting for your feedback. As your evaluation weighs significantly in the final decision, we truly appreciate your response before the discussion period closes on November 18.

Reviewer_CCt2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

TSA predicts a slot activation score that gates the slot-state update and the decoder attention. Activation is predicted from a GRU memory and regularized by usage and sparsity terms. The paper reports large performance gain on YouTube-VIS HQ and OVIS.

Critical strengths

This method is simple and clear.

Evaluation on OVIS with HOTA/IDF1 is useful.

The performance gain is very significant.

Critical weaknesses

The gains do not support the claimed mechanism

They may mainly come from reducing the effective number of slots.

  • LusageL_{\text{usage}} directly penalizes active slots and gives the main gain (Tab. 4 middle: FG-ARI 31.6 -> 61.0). Using gate alone stays near baseline (Exp. #1, #2).
  • With the full regularizer, about 69% of activations are below 0.1 (Fig. 6). With K = 7, only about two effective slots remain per frame. Baselines use all K slots. Slot reduction is already known to improve grouping and reduce fragmentation (SOLV, AdaSlot). The low baseline IDF1 of 1.4-4.3 on OVIS with K = 22 also suggests heavy fragmentation.
  • λreg\lambda_{\text{reg}} controls the effective slot count and is tuned for each dataset (Tab. 7). Its value can range from 0.02 to 0.15***. HOTA varies from 33.1 to 46.5 across its range (Tab. 8), while baseline K stays fixed.
  • Tab. 3 contradicts the occlusion explanation. The largest gains occur for never-occluded objects (T∆ = 0: 7.2 -> 36.6 HOTA, 2.6 -> 38,9 IDF1). The gains shrink with longer occlusion (T∆ > 20: +20.0 HOTA, +21.1 IDF1). If the method mainly preserves identity through occlusion, the trend should be opposite.

So, the central claim or the core contribution is not supported. Tab. 3 shows the opposite trend, yet Tab. 4 link the main gain to slot-count reduction. More controled experiments should be provided. Or re-phrase the key claims.

TSA closely overlaps with Previous work DSSA

  • Both identify that weakly matched or occluded slots absorb other objects' tokens and break slot-object correspondence.
  • DSSA keeps these slots "silent" through competition-modulated aggregation.
  • It uses a per-slot GRU state to preserve identity through occlusion.

The claim that no prior method decided whether a slot is updated or decoded (Sec. 2, App. A, Tab. 5) is too strong. The authors should provide explicit discussions and direct comparisons about these two methods.

Besides, there are some contradictory results in TSA Table 10 and DSSA Table 2. The top-1 values surged from 19.9/26.1 in DSSA to 85.8/90.5 in TSA, although those values are for the same two baseline methods. This makes the reported results not that convincing -- Could there be some unintentional implementation differences or errors?

Circular evidence.

Figure 3 and 7 measure state variance and drift. But Eq. 7 freezes these states by design. Lower values are expected from using the gate itself. They do not show that a certain slot binds back to the same object after it re-appears.

Reproducibility.

The gains are unusually large for a lightweight gate (OVIS IDF1: 3.9 => 36.9; YouTube-VIS HQ: 19.3 -> 46.7). But, the source code is only promised upon publication. These results are needed to be verified during the review. Please provide anonymized code, or ideally further with pre-trained weights, including those for re-evaluated baselines.


  • DSSA: Dual-State Slot Attention: Decoupling Appearance and Identity for Video Object-Centric Learning. ACM MM 2026.
  • SOLV: Self-supervised object-centric learning for videos. NeurIPS 2023.
  • AdaSlot: Adaptive slot attention: Object discovery with dynamic slot number. CVPR 2024.
Questions for authors

The classification accuracy and the number of matched objects are actually a trade-off. You can always obtain high topk scores by matching less number of objects. Just like this, the method achieved marginally higher topk-1/top-3 scores, ~1% higher, while matching must less objects, ~12% less.

8 comments
AuthorsOpenReview ↗

Response to Reviewer CCt2 (Part 1/4)

We thank the reviewer for recognizing the value of our work and for raising these insightful points, which helped us clarify our contributions. We address each concern below.

W1. The gains do not support the claimed mechanism

W1.1. LusageL_{\text{usage}} is a regularizer that helps our mechanism work as designed. Our main contribution is an activation gate: a single activation score that decides, at each frame, which slots are updated from the current frame (Eq. 7) and which take part in decoding (Eq. 8). This gate needs a regularization signal because TSA is trained without visibility labels. Reconstruction alone favors keeping every slot in use. This is harmful when an object disappears, because its slot is then overwritten by unrelated content. LusageL_{\text{usage}} counterbalances reconstruction so that the gate learns to protect a slot when it does not help explain the current frame. However, LusageL_{\text{usage}} does not help on its own. Adding LusageL_{\text{usage}} to three baselines on OVIS improves their HOTA and IDF1 by at most +0.8:

Table. R1. Adding LusageL_{\text{usage}} to the baselines on OVIS.

FG-ARImBOHOTAIDF1
SlotContrast24.116.16.51.4
SlotContrast + LusageL_{\text{usage}}26.116.06.51.5
RandSF.Q29.718.07.32.7
RandSF.Q + LusageL_{\text{usage}}31.719.48.13.5
SmoothSA34.020.38.43.8
SmoothSA + LusageL_{\text{usage}}35.320.48.64.0
TSA61.134.439.438.9

Therefore, the gain comes from combining LusageL_{\text{usage}} with our gating mechanism. LusageL_{\text{usage}} only provides the training signal, and the gate turns it into protection of the slot states.

W1.2. Reducing the number of slots is not beneficial by itself. Table. 4 (left) shows this directly, with all variants trained with the same LregL_{\text{reg}}. When the score only reduces how many slots take part in decoding (#2), the metrics stay at baseline level (30.0 vs. 30.5 FG-ARI, which directly penalizes fragmentation, and 7.9 vs. 7.4 HOTA). When the score gates the state update while all KK slots still take part in decoding (#3), FG-ARI doubles (30.5 → 61.5) and HOTA rises to 21.2. The gain of TSA therefore cannot be explained by using fewer slots alone, but by controlling how the slots evolve over time.

W1.3. λreg\lambda_{\text{reg}}. λreg\lambda_{\text{reg}} controls the strength of the regularizer and affects how well the gate is trained, but the advantage of TSA over the baselines is not a result of tuning it. All methods, including TSA, use the same fixed KK per dataset. Table. 8 serves as a stress test for λreg\lambda_{\text{reg}}, sweeping it over a 5 ×\times range that includes both extremes (from 0.05 to 0.25). Even under this test, TSA stays well above the best baseline on YouTube-VIS HQ, with its worst setting still reaching 33.1 HOTA and 30.5 IDF1, compared with 23.8 and 19.3 for the best baseline. λreg\lambda_{\text{reg}} thus affects how much TSA improves, but not whether it outperforms the baselines.

AuthorsOpenReview ↗

Response to Reviewer CCt2 (Part 2/4)

W1.4. Table. 3 and the occlusion claim. Keeping the identity of an object through a long absence is a well-known open problem in computer vision, and it remains hard even for supervised methods, whose performance also degrades under longer or heavier occlusion (Qi et al., 2022; Shuai et al., 2022; Hong et al., 2025). The objective of TSA is to preserve the slot of an absent object. The slot keeps its previous state and avoids explaining unrelated visible content, so that it can bind back to its object when the object reappears. This addresses a key part of the problem. Full re-identification also depends on how much the object itself changes while it is absent, which remains challenging for every method, especially without labels. It is therefore natural that the performance of TSA, like that of the baselines in Table. 3, decreases as TΔT_\Delta grows. Below, we show that objects indeed change during an absence, and that TSA still clearly outperforms prior methods at every absence length.

Gains for never-occluded and occluded objects. Under unconditional propagation, slots with no visible object to represent are still updated and decoded, so they compete for visible content. Even an object that stays visible is then under constant competition from these slots, so the slot that covers it can change from frame to frame. TSA protects such slots, so a continuously visible object keeps the same slot in every frame. This directly raises HOTA and IDF1, which explains the large gain at TΔT_\Delta = 0. For occluded objects, we verify that the object itself changes during an absence, especially for the non-rigid objects in OVIS. We measure the median cosine similarity of mask-pooled DINOv2 features between the last frame before and the first frame after each absence:

Table. R2. Median cosine similarity of object features before and after an absence on OVIS.

TΔT_\Delta01–1011–20>20
Median cosine similarity0.9370.7970.7830.745

The object changes substantially once it is absent, and the change tends to grow with longer absences. Even a perfectly preserved state therefore matches the object less well at reappearance.

What matters is how much TSA improves over prior methods in this harder setting. Table R3 recalls the results of Table 3 for the two strongest baselines, with the gains of TSA over the best baseline in parentheses.

Table R3. HOTA and IDF1 on OVIS by absence length TΔT_\Delta. Gains are over the best baseline.

HOTA:

MethodTΔT_\Delta = 01–1011–20>20
RandSF.Q6.24.33.62.9
SmoothSA7.24.74.03.7
TSA36.6 (+29.4)30.3 (+25.6)27.5 (+23.5)23.7 (+20.0)

IDF1:

MethodTΔT_\Delta = 01–1011–20>20
RandSF.Q1.81.00.70.5
SmoothSA2.61.31.10.9
TSA38.9 (+36.3)31.6 (+30.3)29.8 (+28.7)22.0 (+21.1)

TSA outperforms the baselines by a large margin at every absence length. Even for absences longer than 20 frames, it reaches 23.7 HOTA and 22.0 IDF1, compared with at most 3.7 HOTA and 0.9 IDF1 for the baselines, a gain of +20.0 HOTA and +21.1 IDF1 without any supervision.

References

[1] Qi, J., Gao, Y., Hu, Y., Wang, X., Liu, X., Bai, X., ... & Bai, S. (2022). Occluded video instance segmentation: A benchmark. International Journal of Computer Vision, 130(8), 2022-2039.

[2] Shuai, B., Bergamo, A., Buechler, U., Berneshawi, A., Boden, A., & Tighe, J. (2022, October). Large scale real-world multi-person tracking. In European Conference on Computer Vision (pp. 504-521). Cham: Springer Nature Switzerland.

[3] Hong, L., Liu, Z., Chen, W., Tan, C., Feng, Y., Zhou, X., ... & Zhang, W. (2025). Lvos: A benchmark for large-scale long-term video object segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence.

AuthorsOpenReview ↗

Response to Reviewer CCt2 (Part 3/4)

W2. TSA closely overlaps with the previous work DSSA

We thank the reviewer for pointing out the previous work DSSA. At the time of our ICLR submission in September 2026, DSSA was available only as an arXiv preprint, and it appeared in the ACM MM 2026 proceedings only in November 2026, so it was not included in our comparison. Nevertheless, we confirm that the two works are different. DSSA splits each slot into two states, a local appearance state and an identity state, and handles them separately. TSA keeps a single state per slot and instead learns how this slot should participate at each frame: whether it is updated and whether it is decoded.

The two works address different problems. DSSA addresses an artifact within Slot Attention at each frame: the attention of each slot is renormalized over tokens before aggregation, so even a weakly matched slot aggregates a full weighted average of tokens, most of which belong to other objects. TSA addresses a problem over time: the slot of an object that is absent from the current frame is overwritten by unrelated content through two pathways, the state update and decoder participation. DSSA thus addresses how slots aggregate tokens within a frame, while TSA preserves the identity of each slot across frames. The two methods consequently act at different points of the pipeline:

Table R4. Conceptual comparison between DSSA and TSA.

DSSATSA
ObjectivePrevent weakly matched slots from absorbing tokens of other objects within a framePreserve the identity of each slot across frames, including when its object is absent
Slot representationTwo states per slot (local appearance and identity)A single state per slot
Where suppression actsToken aggregation inside Slot Attention (CMA)Slot state update and decoder participation
DecisionImplicit, determined by the current-frame attention; not learnedLearned per-slot, per-frame activation score αk,t\alpha_{k,t}, conditioned on the candidate and a temporal memory
State of an absent object's slotLocal state recomputed every frame from attenuated input; identity state updated by a GRU every frameAnchored to its previous state via activation-gated updating (Eq. 7)
DecoderAll KK slots decoded at every frameSlots with a low score do not take part in decoding (Eq. 8)

Direct comparison. To address the reviewer's request, we compare TSA with the results reported by DSSA on the shared metrics:

Table R5. Comparison with DSSA on MOVi-C and YouTube-VIS HQ.

MOVi-C FG-ARI ↑\uparrowMOVi-C mBO ↑\uparrowYT-VIS HQ FG-ARI ↑\uparrowYT-VIS HQ mBO ↑\uparrow
DSSA67.6 ± 2.729.9 ± 3.770.1 ± 1.545.8 ± 1.2
TSA75.1 ± 0.4 (↑\uparrow +7.5)29.9 ± 1.176.3 ± 1.2 (↑\uparrow +6.2)54.2 ± 1.2 (↑\uparrow +8.4)

Object-recognition numbers (Table 10). The baseline numbers in our Table 10 are those reported in the latest version of RandSF.Q, updated in April 2026, whose protocol we follow:

  • In the latest version of RandSF.Q, SlotContrast reaches 85.8 / 95.8 and RandSF.Q reaches 90.5 / 97.9 in Top-1 / Top-3 accuracy.
  • In an earlier version of RandSF.Q, SlotContrast reaches 19.9 / 49.1 and RandSF.Q reaches 26.1 / 60.9 in Top-1 / Top-3 accuracy. These are the same numbers that appear in DSSA.

We evaluate TSA with the released RandSF.Q evaluation code under the latest protocol. The discrepancy therefore reflects the two versions of the reference results, not a difference in our implementation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Response

We thank all reviewers for their detailed feedback. Below we summarize, for each reviewer, the main concerns and how we addressed them. We have also uploaded a revised paper, and we release anonymized code, checkpoints and reproduction scripts at https://anonymous.4open.science/r/tsa-951357/.

Reviewer CCt2

  • Whether gains come from using fewer slots. LusageL_{\text{usage}} is a regularizer that supports our main mechanism, since the activation score must be learned without labels. LusageL_{\text{usage}} brings no gain on its own: adding it to the baselines barely changes their results (Table R1 in Response to CCt2). The gain appears only when LusageL_{\text{usage}} is combined with our mechanism, which controls how each slot is updated and decoded over time.
  • Whether TSA handles long occlusion (Table 3). Re-identification after a long absence is harder for every method, including supervised trackers, as objects may change during an absence. TSA still clearly outperforms the strongest baseline, SmoothSA, at every absence length, both for never-occluded objects (36.6 vs. 7.2 HOTA at TΔT_\Delta = 0) and for the longest absences (23.7 vs. 3.7 HOTA and 22.0 vs. 0.9 IDF1 for TΔT_\Delta > 20).
  • Relation to DSSA. DSSA and TSA are two different methods addressing temporal slot learning. While DSSA splits each slot into an appearance state and an identity state and suppresses weakly matched slots during token aggregation within a frame, TSA keeps a single state per slot and learns, at each frame, whether the slot is updated and whether it takes part in decoding, so that the identity of each slot is preserved across frames. TSA outperforms DSSA on the shared metrics, and the baseline numbers in Table. 10 follow the latest version of RandSF.Q.
  • Circular evidence (Fig. 3, Fig. 7). Eq. 7 only allows a state to be protected, while when it is protected is learned without visibility labels. The two figures therefore show learned behavior and confirm that TSA works as intended. We further show identity switches around occlusion events, which confirm that TSA binds reappearing objects back to the same slot far more often than the baselines (Table R6 in Response to CCt2).
  • Accuracy vs. number of matches (Table 10). We evaluate with a stricter matching threshold, under which a slot must largely overlap its object. TSA then matches the most objects and still achieves the best accuracy, so its accuracy does not come from matching fewer objects (Table R7 in Response to CCt2).

Reviewer GuiX

  • Dataset-specific hyperparameters. Setting hyperparameters per dataset is standard practice in unsupervised object-centric learning, and the advantage of TSA is independent of these values. TSA is stable across a broad range on two datasets (Table. 8 in the paper; Table. R1-R2 in Response to GuiX), and a single setting shared across all benchmarks still outperforms the baselines on the real-world datasets (Table. R3 in Response to GuiX).
  • Number of slots. We clarify that TSA controls how each slot evolves over time rather than the number of slots. We also vary the slot budget, and TSA remains above the best baseline with larger budgets (Table. R4 in Response to GuiX).
  • Comparison with SlotCurri. We did not include SlotCurri in the current version, as SlotCurri addresses over-fragmentation, where one object is split across several slots, while TSA aims to preserve the identity of each slot across frames, including when its object is absent. SlotCurri also follows a different evaluation protocol. We also reproduce SlotCurri under our protocol, and TSA clearly outperforms it (Table. R5 in Response to GuiX).

Reviewer qoiE

  • Identity preservation through absence. As requested, we measure the identity switch (IDSW) rate around occlusion events on correctly segmented objects (Table. R1 in Response to qoiE) and decompose HOTA into detection accuracy (DetA) and association accuracy (AssA) (Table. R2 in Response to qoiE). TSA switches identity far less often than the baselines (31.8% vs. 76.5% IDSW rate for SmoothSA on OVIS), and it improves both DetA and AssA (on OVIS, 27.2 vs. 5.0 DetA and 56.2 vs. 14.7 AssA over SmoothSA).
  • Shared vs. separate scores. We compare TSA with a variant that uses separate scores for the two pathways. This variant performs worse on all metrics, which supports our design of a single shared score.
  • Scale invariance of Eq. 8. The training objective discourages all scores from becoming low in the same frame, so slots of visible objects remain active and a slot with a low score does not take part in decoding.
  • Conclusions of Sec. 3. To address the reviewer's concern, we clarify which statements in Sec. 3 rely on assumptions and link each failure pathway to its experimental evidence. We have revised Sec. 1 and Sec. 3 accordingly.
AuthorsOpenReview ↗

Revised Manuscript

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript, in which all changes are highlighted in blue. The main additions are:

Main text

  • Abstract, Sec. 5.3 and Conclusion: We clarify that the gains of TSA come from both continuously visible objects and objects that disappear and reappear, rather than from occlusion handling alone.
  • Sec. 1 and Sec. 3: We clarify which statements rely on assumptions and link each failure pathway to its experimental support (App. G.2 for Pathway I, Sec. 5.4 for Pathway II).
  • Sec. 5.3, Table 4: New analysis of identity switches around occlusion events on OVIS.
  • Sec. 5.4, Table 5(d): New ablation comparing the shared activation score with separate scores for the two pathways on OVIS.
  • Limitations: We acknowledge that the regularization weights are set per dataset, so applying TSA to a new dataset may require adjusting them to its characteristics.

Appendix

  • App. E, Tables 10 and 11: New sensitivity analysis on MOVi-C and results with a single hyperparameter setting shared across benchmarks.
  • App. F, Table 12(d): The shared vs. separate score ablation on YouTube-VIS HQ.
  • App. G.3, Table 13: Object-recognition results at a stricter matching threshold (IoU > 0.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.