acceptodds
Withdrawn from ICLR 2027

Hold the Evidence, Then Look Back: Resolving Identity Conflicts in Memory-Based Visual Object Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory-based visual object trackers built on promptable segmentation models still confuse identities when similar objects overlap, occlude one another, or return after long absences. These trackers typically decide such conflicts in the frame where they appear, although the overlap that reveals a conflict usually follows the identity error and the evidence that resolves it arrives during and after the conflict. We present PCIA-Track, a training-free, plug-in framework built on one principle: keep the evidence during a conflict and decide the identity after it. Specifically, Hold-and-Report (HaR) holds the memory of a conflicting track but keeps reporting its prediction, and relocates the target among class-agnostic proposals when that prediction duplicates another object. Post-conflict identity arbitration (PCIA) waits until the tracks separate, replays the buffered video backward from two fresh, memory-free probes, and hands the identity over only when the candidate, and not the incumbent, reconnects to the incumbent's last stable window. A small-target confidence floor (SCF) removes spurious outputs of vanished small targets. PCIA-Track improves on SAM 3 on LVOSv2, LVOSv1, VOST, DiDi and LaSOT_ext, where it is the best of the compared trackers, and reaches a quality score of Q = 0.735 on VOTS2026, above the best entry of the challenge. It keeps the model weights and the architecture unchanged.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual object tracking, video object segmentation, SAM 3, identity conflicts, memory-based tracking, training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_orhKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PCIA-Track addresses identity confusion in memory-based visual tracking by delaying identity decisions until evidence becomes clearer after a conflict. It preserves the track's memory while continuing to report predictions, then uses backward replay to check whether a competing track reconnects to the target's earlier stable history. The framework augments an existing segmentation-based tracker without additional training. Benchmark comparisons, component ablations, and analyses of individual arbitration outcomes report overall tracking gains while showing that the benefit of retrospective identity correction varies across datasets.

Critical strengths
  • The distinction between preserving predictions during a conflict and deciding identity afterward is well motivated and operationally precise (Sections 3.3–3.4). Table 12 supports reporting with duplicate suppression: +0.136 Q-points, with a positive bootstrap interval, relative to withholding all held outputs.
  • The paper provides unusually transparent analysis: component removals, evaluator and mask-to-box checks, and explicit failed hand-overs (Appendices A–E). The complete system improves over SAM 3 on all five benchmarks in Table 1 and reaches 0.7352 Q on VOTS2026.
Critical weaknesses
  • The central arbitration mechanism has limited demonstrated reliability. Only two of four LVOSv2 hand-overs and one of two DiDi hand-overs are correct; its entire VOTS2026 gain comes from one sequence, and it lowers DiDi Q by 0.37 points (Section 5, Appendix E). Incorrect reference histories also defeat recovery when a correct candidate exists. These results support occasional useful correction more strongly than broadly reliable identity resolution.
  • The headline gains compare against plain SAM 3, whereas the implementation inherits SAFEDAM's memory and candidate machinery (Section 3.1). Table 2 removes components individually but does not report the identical base with all three additions disabled; Table 3's published SAFEDAM result does not supply that controlled comparison. Consequently, the combined contribution of the proposed additions is not cleanly separated from the inherited tracker.
Questions for authors
  1. Clarification: What are the results of the identical implementation with SCF, HaR, and PCIA all disabled, under the same evaluation protocols?
Reviewer_7ahmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses identity swaps in memory-based visual object trackers built on SAM 3, which happen when similar objects overlap, occlude one another or reappear. It argues that trackers settle these conflicts too early: the overlap that reveals a conflict usually comes after the identity has already jumped, and the evidence that resolves it arrives during and after the conflict. Its principle is to keep the evidence during a conflict and decide the identity afterward.

Contributions.

Hold-and-Report (HaR): when a conflict check freezes a track’s memory, the track keeps reporting its own prediction instead of an empty mask. If that prediction duplicates another target, HaR relocates the target among SAM 3’s automatic mask proposals, scored by the frozen memory. Post-conflict identity arbitration (PCIA): after the incumbent and a candidate track separate, two fresh memory-free tracker copies, one from each current mask, are run backward through the buffered frames to the incumbent’s last stable window. The identity is handed to the candidate only if the candidate’s probe links back and the incumbent’s does not. Small-target confidence floor (SCF): removes tiny, low-confidence masks left behind by objects that have vanished. Training-free plug-in: all of this sits on the SAFEDAM code (SAM 3 with a DAM4SAM-style memory, candidate tracks and a conflict check), with no retraining.

Validation.

Main results: gains over SAM 3 on LVOSv2 (89.9 J&F), LVOSv1, VOST, DiDi and LaSOText, and Q = 0.735 on VOTS2026, above the challenge winner, without a vision-language model. Component analysis: each component removed from the full system on LVOSv2, VOTS2026 and DiDi. Deeper analyses: conflict statistics and output rules for held tracks, with bootstrap intervals per-benchmark arbitration outcomes and failure analysis case studies (e.g., three shuffled identical cups) cost: memory and runtime

Critical strengths

It targets an important, unsolved failure mode with a principled fix. Identity swaps between similar objects remain one of the main failure modes of today’s best SAM-based trackers, and per-frame decisions can’t fix them by design, because the overlap that reveals a conflict comes after the identity has already jumped. The paper’s analysis on LVOSv2 supports this: held tracks have already jumped at detection, often return to the target later in the hold, and the true target often survives as a candidate track (Fig. 3). Its answer, to keep the evidence during a conflict and decide afterward from memory-free backward probes with a conservative one-sided rule, is a genuinely different way of thinking about identity in memory-based tracking. The cup-shuffling case, where PCIA restores an identity that SAM 3, SAMURAI and DAM4SAM all lose, shows it solving exactly the cases that matter. Exceptionally careful and transparent evaluation. Evaluator versions, mask-to-box conventions and every baseline source are documented. Thresholds were selected with a rule fixed in advance. Every wrong keep and wrong hand-over is traced to its cause. The limitations section openly reports where PCIA hurts (DiDi) and its memory and runtime overhead. I spot-checked the cited baseline numbers against their sources and they match. This level of care makes the results trustworthy as reported and the paper easy to build on.

Critical weaknesses

The gain over SAM 3 is not yet attributed to the proposed components. On LVOSv2, the full system beats SAM 3 by 1.6 J&F, but removing HaR, PCIA and SCF individually costs only 0.22, 0.11 and 0.03. Removing the conflict check entirely still leaves the tracker 1.1 above SAM 3. On VOTS2026, SAFEDAM, the code the method is built on, already scores 0.722, and the component removals explain only about 0.3 of the 1.3-point gap to 0.735. On LaSOText (+3.4 AUC), relocation never runs and there is no ablation. As written, the abstract credits the method with gains that largely belong to the base tracker. I expect this is fixable: adding the base tracker to the main tables and explaining the VOTS2026 gap would let readers see what the method itself contributes. The components’ effects are small and rest on few events. PCIA hands over just seven times across three benchmarks, and three of those hand-overs move the identity onto the true target. Its VOTS2026 gain comes from one sequence and its LVOSv2 gain essentially from one video. On DiDi it lowers Q by 0.37, and with S = 5 on LVOSv2 it makes no hand-overs at all. No run-to-run variance is reported, so differences of 0.02–0.2 points can’t yet be separated from noise. This is partly inherent, since hard identity conflicts are rare, and the authors acknowledge it. But variance estimates are needed to show the effects are real.

Questions for authors

What does the base tracker (SAFEDAM without SCF, HaR and PCIA) score on every benchmark in Table 1? What explains the gap between SAFEDAM’s 0.722 on VOTS2026 and the 0.735 of the full system, given that the component removals account for only about 0.3 of it? What is the run-to-run variance on LVOSv2, DiDi and VOTS2026? How many submissions were made to the VOTS2026 server, and did any configuration choices follow server feedback?

Reviewer_arjTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper presents PCIA-Track, a training-free framework for improving SAM3 on the problem of identity conflicts in object tracking. The idea is that when the identity conflict is potentially happening, the tracker follows two tracks of the target and the candidate, waits until the objects separate, and then replies with fresh memory backward to the frames before identity conflict to determine the correct target trajectory to keep following. PCIA integrated to SAM3 leads to improved performance on several datasets.

Critical strengths

Strength 1: The idea is well motivated for improving object tracking during identity conflicts. PCIA-Track shows approximately 5% improvements on LVOSv1 and LaSOText datasets.

Critical weaknesses

Weakness 1: Limited research novelty and evidence that the model works. The model is largely based on the challenge solution from VOTS2026, with some known engineering components like small-target confidence floor component. The differences in performance between SAM3 and PCIA-Track on LVOSv2, VOST, and DiDi (which includes distractors in all sequences, thereby could have potentially more identity switches) are marginal. Authors acknowledge in the limitations that the hand-overs are rare, which hinders the importance of the proposed approach, if the underlying issue is already solved by existing trackers. If PCIA-Track was resolving identity shifts of SAM3, the expected increase in results would be in robustness metric, but on VOTS2026 and DiDi (Table 3 and Table 9), robustness is almost the same as for SAM3.

Weakness 2: Not fair evaluation setup. PCIA-Track is integrated on top of SAM3, with DAM4SAM memory management protocol. Yet, the result for DAM4SAM in Table 1 are present for SAM2.1 base. The same holds for some other models, including SAMURAI and SAM2Long. For fair evaluation, it would be better to implement PCIA on top of SAM2.1 and evaluate all models with SAM2.1 base and / or re-evaluate all baselines with SAM3 base. Also, SAFEDAM + SAM3 or DAM4SAM + SAM3 (parts of PCIA-Track) are not evaluated in Table 1 nor 2.

Questions for authors

Question 1: Could you show, by some experiments, that the issue of identity shifts has not already been resolved by the SAM3 (+ DAM4SAM) combination?

Question 2: Can you evaluate related works from Table 1 and Table 3 that were ran on top of SAM2 with the SAM3 base, or implement PCIA-Track on SAM2 (or 2.1) base and evaluate on these datasets?

Question 3: Can you include ablation for SAFEDAM + SAM3 and DAM4SAM + SAM3 which are the parts of the constructed PCIA-Track?

Question 4: What happens when there are more than 2 objects in the scene (for example, check sequences in UTB180[1] dataset)?

[1] UTB180: A High-quality Benchmark for Underwater Tracking, Alawode, Basit and Guo, Yuhang and Ummar, Mehnaz and Werghi, Naoufel and Dias, Jorge and Mian, Ajmal and Javed, Sajid; ACCV 2022

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.