acceptodds
Under review as a conference paper at ICLR 2027

SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

Abstract

Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-in-First-out rule, allowing temporal consistency in complex video sequences. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of the 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual object tracking, reinforcement learning, memory control, Segment Anything Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LNGUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The study presents a general, data-driven approach that frames the essential memory mechanism of the SAM family of models as a sequence decision-making task. Extensive experiments show that the approach improves SAM performance and generally outperforms hand-crafted rules across a broad range of benchmarks for visual object tracking and video object segmentation.

Critical strengths
  1. The approach is well motivated and clearly formulated. The memory mechanism is an important component of SAM family of models. Learning to manage memory is well motivated for SAM-based video trackers as decisions about which observations to retain are critical to the predictions of later frames. The framework including controller inputs, action space, reward, and training procedure are clearly described, which makes the approach straightforward to follow. The proposed light controller that improves the use of a limited memory budget could be useful in practice and provide foundation for future work on adaptive memory management in SAM.

  2. The experiments cover a broad range of benchmarks and demonstrate solid improvements. The experiments cover a substantial range of tracking and video object segmentation benchmarks and evaluate both SAM2.1 and SAM3, showing the robustness of the framework across different backbones. The experiment results show compelling improvements over SAM3 and competing methods that use hand-crafted memory rules on several challenging benchmarks.

Critical weaknesses
  1. The controller design needs further justification. A fixed budget and a lightweight MLP are reasonable choices. However, the paper does not sufficiently explain why the bank must remain full after initialization under the single-entry eviction formulation. For example, does this design offer advantages over allowing the controller to retain a variable number of memories within the same maximum capacity when some memories are unreliable or redundant?
  2. The feature ablations do not isolate specific representation choices. Appendix E.1 provides useful feature-group ablations but does not separately examine foreground mean pooling with alternative pooling methods or evaluate alternatives to the chosen pairwise similarity measures.
Questions for authors
  1. Can you explain the intuition behind keeping the memory bank full after initialization? Have you compared single-entry eviction with allowing the controller to retain a variable number of memories within the same maximum capacity, particularly when some memories are unreliable or redundant?
  2. Have you compared foreground mean pooling with alternative pooling methods, while keeping the other components fixed?
  3. Have you evaluated alternative measures for computing similarities between memories, beyond the feature-group ablations in Appendix E.1?
  4. In Table 4, do SAM3 ours and SAM3RL use the same auxiliary-pointer selection rules and temporal encoding? Can you clarify whether the learned memory-selection policy is the only difference. If not, could you provide a comparison while the other settings are fixed?
2 comments

Thank you for your careful review, and for finding our approach well motivated, our controller practical, and our benchmark coverage broad. Below we answer each of the weaknesses and questions you raised.

W1 / Q1 — Full bank vs. variable number of memories

We have not compared against a variable-sized bank. We keep the bank full mainly for consistency with prior work. The native rules of SAM 2 and SAM 3 also evict exactly one entry per update, and prior rules such as DAM4SAM, SAMURAI and SAM2Long likewise keep the bank full. A full bank also keeps the design simple, as each action maps to exactly one slot and parallel rollouts can be batched without padding or masking. Even with a full bank, unreliable or redundant memories can be handled, since the policy can evict any stored frame or reject the incoming one at every step. Whether a variable-sized bank could still help, for example by keeping only a few reliable memories, is an interesting question for future work. We added this justification to Section 3 and mention a variable-sized bank as future work in Section 6.

W2 / Q2 + Q3 — Foreground mean pooling and similarity measures

Foreground mean pooling. We did not train actors with alternative pooling methods. We preferred foreground pooling over whole-frame pooling, as it keeps the memories more distinguishable. Specifically, on our LaSOT validation split, the mean cosine similarity between memory slots is 0.99 with whole-frame pooling but 0.87 with foreground pooling, likely because the mostly unchanged background dominates the small changes of the object. Feeding the unpooled features instead would require a much heavier actor, which we avoid to keep the controller lightweight. We agree this choice deserved more explanation, which we added to Section 3.1 and Appendix E.1.

Similarity measures. We did not try any other similarity measures. We use cosine similarity, as it is the standard measure for comparing embeddings. We expect other measures to matter little, so we did not consider them worth a full ablation. If you consider it important, we are happy to add such an ablation to the camera-ready version.

Q4 — Gains attributable to the learned eviction

SAM 3 (ours) in the submitted Table 4 indeed used SAM 3’s native object-pointer selection and pointer temporal encoding, not those of SAM3RL. This is because Table 4 was designed as a stride ablation, in which SAM 3 (ours) served as a minimal fix of the official strided SAM 3. The official version collapses because it reselects a shifted bank at every frame, likely a setting SAM 3 never sees in training. The fix keeps the bank unchanged between updates, as our method does, and changes nothing else. To avoid confusion, this row is now called SAM 3 (fix).

We agree that isolating the learned eviction is important. The name SAM (ours) now denotes a new row, which runs SAM 3’s eviction rule inside our MDP (Section 3.1). It shares the stride, the recently evicted pointer rule and the pointer temporal encoding with SAM3RL, so only the eviction policy differs. SAM3RL outperforms SAM 3 (ours) on all seven benchmarks, by 0.9 on average and most on VOT-LT2022 (+2.2) and VOTS2026 (+1.3). Going from SAM 3 (fix) to SAM 3 (ours), which changes only the pointer rule and encoding, adds 0.6 on average. Notably, SAM 3 (ours) alone already outperforms all prior methods on VOTS2026, VOTS2025 and LaSOT, but still falls short of our learned policy. We updated Table 4 and Section 5 accordingly.

ModelsFPSVOTS2026 (Q)VOTS2025 (Q)DiDi (Q)VOT-LT2021 (F)VOT-LT2022 (F)LaSOT (AUC)LaSOT-ext (AUC)
SAM 3115.366.473.471.477.162.875.961.7
SAM3RL114.969.3 (+2.9)75.4 (+2.0)71.7 (+0.3)77.2 (+0.1)65.6 (+2.8)77.1 (+1.2)65.2 (+3.5)
SAM 3615.555.5 (−10.9)66.9 (−6.5)60.5 (−10.9)76.2 (−0.9)61.8 (−1.0)72.0 (−3.9)57.4 (−4.3)
SAM 3 (fix)615.868.1 (+1.7)75.9 (+2.5)71.2 (−0.2)75.7 (−1.4)65.6 (+2.8)77.4 (+1.5)64.8 (+3.1)
SAM 3 (ours)616.168.7 (+2.3)76.6 (+3.2)71.8 (+0.4)77.9 (+0.8)66.1 (+3.3)77.8 (+1.9)64.2 (+2.5)
SAM3RL616.070.0 (+3.6)76.9 (+3.5)72.0 (+0.6)79.0 (+1.9)68.3 (+5.5)78.4 (+2.5)64.7 (+3.0)
Reviewer_LNGUOpenReview ↗

Response to Authors' Comment

Dear Author,

Thank you for your clarifications and for revising the manuscript. I have carefully read the revised version. The revisions make the paper clearer. I continue to find the work well motivated and technically sound, and I am happy to maintain my positive assessment.

Best regards

Reviewer_3HUfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SAM3RL, a learned memory controller for a frozen SAM 3 tracker. A small MLP trained with PPO on LaSOT decides which memory to evict, including whether to reject the incoming frame. The policy is combined with sparse memory updates and an action-dependent object-pointer selection rule. SAM3RL ranks first in ten of the twelve main benchmark comparisons, covering tracking and video object segmentation. A separately trained SAM2RL applies the same training recipe to SAM 2.

Critical strengths

1.The controller generalizes beyond individual training videos. This is the main advance over the video-specific RL policies discussed in Section 2. The backbone remains frozen, and a policy trained with bounding-box supervision also improves mask-based segmentation. These results make a useful case for learning memory selection rather than adding another task-specific rule. 2.The evaluation provides convincing evidence of improvement. Against the stride-matched SAM 3_ours baseline, SAM3RL still gains 1.9 points on VOTS2026 and 2.7 on VOT-LT2022 (Table 4). The three-seed ablations and the correction of a race condition affecting several baselines add confidence in the comparisons. The memory visualizations also show both successful recovery and a case where retaining old observations becomes harmful.

Critical weaknesses

1.The gain attributable to learned eviction remains unclear. Table 4 controls the update schedule, but does not clearly state whether SAM 3_ours also shares SAM3RL’s object-pointer selection and temporal encoding. Figure 6 shows that pointer selection affects performance. The results therefore establish the benefit of the complete system more clearly than that of the eviction policy alone. 2.The cross-backbone comparison needs a narrower interpretation. The heuristic baselines retain their SAM 2 hyperparameters when ported to SAM 3, whereas SAM3RL is trained for the new backbone. This establishes an advantage over unchanged ports, but leaves open how much of the gap would remain after modest validation-set tuning. The results against native SAM 2 implementations are reassuring, but do not resolve this concern about the SAM 3 comparison.

Questions for authors
  1. Does SAM 3_ours use the same object-pointer selection and pointer temporal encoding as SAM3RL? If these differ, please provide a comparison in which only the eviction policy changes.
  2. Have the relevant thresholds of SAMITE3 and DAM4SAM3 been tuned on the held-out validation split? A limited sensitivity study would help establish whether their weaker SAM 3 results reflect the selection rules or their calibration.
  3. The temporal-only actor nearly matches the full actor on VOT-LT2022, but falls further behind on MOSEv2 (Table 15). Does this suggest that learned temporal scheduling accounts for much of the tracking gain, while content cues matter more for segmentation?
2 comments

Thank you for your careful review, and for finding our evaluation convincing, our comparisons reliable, and our visualizations useful. Below we answer each of the weaknesses and questions you raised.

W1 / Q1 — Gains attributable to the learned eviction

SAM 3 (ours) in the submitted Table 4 indeed used SAM 3’s native object-pointer selection and pointer temporal encoding, not those of SAM3RL. This is because Table 4 was designed as a stride ablation, in which SAM 3 (ours) served as a minimal fix of the official strided SAM 3. The official version collapses because it reselects a shifted bank at every frame, likely a setting SAM 3 never sees in training. The fix keeps the bank unchanged between updates, as our method does, and changes nothing else. To avoid confusion, this row is now called SAM 3 (fix).

We agree that isolating the learned eviction is important. The name SAM (ours) now denotes a new row, which runs SAM 3’s eviction rule inside our MDP (Section 3.1). It shares the stride, the recently evicted pointer rule and the pointer temporal encoding with SAM3RL, so only the eviction policy differs. SAM3RL outperforms SAM 3 (ours) on all seven benchmarks, by 0.9 on average and most on VOT-LT2022 (+2.2) and VOTS2026 (+1.3). Going from SAM 3 (fix) to SAM 3 (ours), which changes only the pointer rule and encoding, adds 0.6 on average. Notably, SAM 3 (ours) alone already outperforms all prior methods on VOTS2026, VOTS2025 and LaSOT, but still falls short of our learned policy. We updated Table 4 and Section 5 accordingly.

ModelsFPSVOTS2026 (Q)VOTS2025 (Q)DiDi (Q)VOT-LT2021 (F)VOT-LT2022 (F)LaSOT (AUC)LaSOT-ext (AUC)
SAM 3115.366.473.471.477.162.875.961.7
SAM3RL114.969.3 (+2.9)75.4 (+2.0)71.7 (+0.3)77.2 (+0.1)65.6 (+2.8)77.1 (+1.2)65.2 (+3.5)
SAM 3615.555.5 (−10.9)66.9 (−6.5)60.5 (−10.9)76.2 (−0.9)61.8 (−1.0)72.0 (−3.9)57.4 (−4.3)
SAM 3 (fix)615.868.1 (+1.7)75.9 (+2.5)71.2 (−0.2)75.7 (−1.4)65.6 (+2.8)77.4 (+1.5)64.8 (+3.1)
SAM 3 (ours)616.168.7 (+2.3)76.6 (+3.2)71.8 (+0.4)77.9 (+0.8)66.1 (+3.3)77.8 (+1.9)64.2 (+2.5)
SAM3RL616.070.0 (+3.6)76.9 (+3.5)72.0 (+0.6)79.0 (+1.9)68.3 (+5.5)78.4 (+2.5)64.7 (+3.0)

W2 / Q2 — DAM4SAM3 and SAMITE3 hyperparameter tuning

We ported all baselines with their original SAM 2 parameters, without tuning them on SAM 3. We agree, however, that a sensitivity study on SAM 3 is needed. We therefore repeated the original sensitivity studies of SAMITE and DAM4SAM on SAM 3. As their authors did for SAM 2, we vary one parameter at a time, over the same parameters and values. We evaluate each value on the same held-out LaSOT validation split as SAM3RL.

The full tuning results are in the new Appendix F. For SAMITE3, no single-parameter change improves validation quality over the defaults (Table 16), so its tuned and default configurations coincide. For DAM4SAM3, single-parameter changes improve validation quality by up to 0.8, but these gains are not additive (Table 17). We therefore also evaluated all 16 combinations of default and improved values, and the single-parameter changes \theta_{area} \= 0.1 and N_M \= 15 remain tied for best. We evaluated these two tuned variants on the test benchmarks below (Table 18).

Metric   Benchmark      SAM 3      SAMITE3      DAM4SAM3      DAM4SAM3      DAM4SAM3      SAM3RL   
default = tuneddefaultθarea=0.1\theta_{area} = 0.1NM=15N_M = 15
QualityVOTS202666.465.4 (−1.0)66.7 (+0.3)67.5 (+1.1)67.8 (+1.4)70.0 (+3.6)
VOTS202573.473.6 (+0.2)73.3 (−0.1)74.2 (+0.8)74.0 (+0.6)76.9 (+3.5)
DiDi71.471.8 (+0.4)71.8 (+0.4)72.3 (+0.9)71.9 (+0.5)72.0 (+0.6)
F-scoreVOT-LT202177.178.3 (+1.2)77.6 (+0.5)77.7 (+0.6)77.4 (+0.3)79.0 (+1.9)
VOT-LT202262.866.2 (+3.4)63.4 (+0.6)62.0 (−0.8)63.6 (+0.8)68.3 (+5.5)
AUCLaSOT75.977.6 (+1.7)77.3 (+1.4)77.6 (+1.7)77.3 (+1.4)78.4 (+2.5)
LaSOT-ext61.764.7 (+3.0)65.0 (+3.3)65.2 (+3.5)65.1 (+3.4)64.7 (+3.0)
J&F / ḞMOSEv259.853.4 (−6.4)58.0 (−1.8)58.9 (−0.9)58.3 (−1.5)60.9 (+1.1)
SA-V val82.981.1 (−1.8)80.4 (−2.5)80.3 (−2.6)80.4 (−2.5)85.4 (+2.5)
SA-V test84.682.2 (−2.4)83.1 (−1.5)82.8 (−1.8)82.7 (−1.9)84.6 (+0.0)
LVOSv288.789.3 (+0.6)88.9 (+0.2)88.5 (−0.2)88.2 (−0.5)89.5 (+0.8)
Mean Δall 11–−0.1+0.1+0.2+0.2+2.3

\theta_{area} \= 0.1, the slightly better of the two, improves DAM4SAM3 on most tracking benchmarks (VOTS2026 +0.8, VOTS2025 +0.9, DiDi +0.5, LaSOT +0.3, LaSOText +0.2, VOT-LT2021 +0.1) and on MOSEv2 (+0.9), but costs 1.4 on VOT-LT2022 and 0.1 to 0.4 on the other three VOS benchmarks. Relative to SAM3RL, tuning changes the order only on DiDi, the distractor benchmark introduced together with DAM4SAM, where DAM4SAM3 with \theta_{area} \= 0.1 now leads (72.3 vs. 72.0). Across all eleven benchmarks, tuning reduces SAM3RL’s average lead over DAM4SAM3 only from 2.2 to 2.1, and both tuned variants remain below plain SAM 3 on all four VOS benchmarks. The weaker SAM 3 results of the ports therefore mainly reflect their design rather than their untuned hyperparameters.

We keep the default configurations in the main tables, as tuning does not change the conclusions, but we are happy to add the tuned variants there as well.

Q3 — Temporal-only vs. full actor

Partially yes. The temporal-only actor recovers most of the gain on the box-annotated tracking benchmarks, staying within 0.5 of the full actor on VOT-LT2022, LaSOT and LaSOT-ext. It falls further behind on the mask-annotated VOTS2026 and MOSEv2, by 0.8 and 1.4. On the other hand, removing only the temporal encodings keeps the actor within seed noise of the full actor on all five benchmarks (Table 14). Temporal encodings are therefore not necessary, and the same gains can be learned from the remaining content-sensitive features. So while single feature groups can give most of the gain on some benchmarks, the full actor, which combines all feature groups, gives the most robust behavior across all of them (Tables 14 and 15).

Reviewer_ukv8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper replaces SAM3's hand-crafted memory rule, a FIFO variant gated on an object-score threshold, with a learned one. A small actor MLP (~3.2M parameters), trained with PPO on a stride-6 (5 FPS) subsample of LaSOT with the backbone frozen, observes temporal, confidence, semantic, and geometric features for all eight slots of the over-full bank and picks one of seven candidates to evict: six stored frames or the incoming one, the latter rejecting the update. The reward is per-frame IoU (1 when both sides report absence) minus a precomputed baseline, namely the same tracker's quality under SAM3's default rule. Inference is greedy, updating memory every s-th frame with s = round(ν/5), slightly faster than SAM3 on 30 FPS benchmarks (16.0 vs 15.3 FPS) and marginally slower when every frame is stored (15.4 vs 15.7 on MOSEv2). SAM3RL improves SAM3 on 11 of 12 benchmarks, ranks first on 10 (VOTS2026 +3.6), and is the only compared method never falling below baseline on the four main VOS benchmarks. SAM2RL, the same recipe on SAM2, improves all 12 and leads on 9 against original author implementations. The paper also ports five SAM2 methods to SAM3, fixes a race condition in the official SAM3 video predictor (all numbers post-fix), and reports three-seed ablations plus a stride-disentangling control.

Critical strengths
  1. The formulation makes an open design question testable. Memory eviction is cast as an MDP over the seven-slot bank, and both native rules turn out to be expressible as policies in that same action space: FIFO always evicts the oldest evictable entry, while SAM3's rule switches between evicting the oldest and the most recently stored entry depending on an object-score-gated IoU threshold. Since the backbone is frozen and the bank is fixed, the eviction decision is the main thing that changes against the base tracker. Two side changes remain, the update stride and the rule that fills the nine object pointers outside the bank, and each is examined separately (the SAM3ours arm in Table 4 for the stride; the four-rule ablation in Appendix E.2 for the pointer rule), which bounds how much either could contribute, even though the equal-stride comparison does not fully separate the pointer rule (see my question to the authors). Most competing methods also touch mask selection, prompting or the number of banks, as the authors themselves point out, so without this setup the argument would rest on leaderboard comparisons against methods that differ in several ways at once. With it, I can see what the learned controller buys and what the surrounding machinery buys.

  2. One recipe, two backbones, unseen videos. The controller is trained once on the LaSOT training split and then used as-is on SAM3, where it improves 11 of 12 benchmarks and tops 10. The same recipe, retrained on SAM2, improves that backbone on all 12 and leads on 9 — and there the comparison runs against the compared methods' own original SAM2 implementations rather than the authors' ports, which is the version of the comparison I trust most. The one earlier attempt I am aware of at RL memory control for SAM2 (Adamyan et al., 2025, arXiv:2507.08548) fit one agent per video, by design, so it demonstrated an upper bound rather than a usable tracker. Getting a single policy to generalize to unseen videos, and getting the same recipe to hold on a newer backbone, is the actual advance here, not a retuning of a published idea. I would add that it is the only compared method that never falls below baseline on the four main VOS benchmarks, which suggests the policy learns something about which frames matter rather than how to score well on one benchmark family.

Critical weaknesses
  1. The SAM3 evidence does not line up with the public challenge results. The transfer-failure claim rests on ports; the paper itself notes that none of the compared methods has an official SAM3 implementation. On VOTS2026 its DAM4SAM3 gains 0.3 over SAM3 (66.7 vs 66.4), while the official workshop figure, labeled VOTS2026, puts the same port at 69.3 against a 64.5 baseline (+4.8) — a gain sixteen times larger. A protocol difference would move both scores alike, yet the paper's baseline sits 1.9 above the official one while its port sits 2.6 below: if the protocol lifts both comparably, the port should land near 71.2, not 66.7. That 4.5-point residual must come from the port itself — either a genuine transfer failure or an untuned port — and the paper separates neither. The sequence set may also differ: Table 8 gives 170 sequences / 375 targets for VOTS2026 versus the official 120 / 271, and 170 = 144 + 26 is exactly the union retaining the 50 sequences the organizers removed. Re-running both on the official set would settle it.

  2. Four of ten first places fall within the seed noise the paper reports. Main tables use the best of three seeds chosen on validation, while Tables 14–15 put three-seed SEM at 0.1–0.8. Four wins lead by 0.8 or less: DiDi +0.2 (over a three-way tie), LVOSv2 +0.2, VOT-LT2021 +0.7, LaSOT +0.8; SA-Vtest is a baseline tie, not a win. The results the argument leans on are far outside that band (VOTS2026 +2.1, VOTS-RT +2.4, VOT-LT2022 +2.1 over runner-up), so my lean holds; but "first on 10 of 12" claims more than the evidence supports, and with all three seeds already trained a three-seed table costs only evaluation time.

Questions for authors

Q1. On VOTS2026, your Table 1 has SAM3 at 66.4 and DAM4SAM3 at 66.7 (+0.3); the official workshop figure — axis label "DAM4SAM3 (SAM2 → SAM3 backbone)" — has SAM3 at 64.5 and a third-party port of the same method at 69.3 (+4.8). Your baseline sits 1.9 above the official one while your port sits 2.6 below it, and absent method-specific sequence effects, a common protocol or sequence-set difference would move both in the same direction. The set may in fact differ: Table 8 lists 170 sequences / 375 targets versus the official 120 / 271, and 170 = 144 + 26 is exactly VOTS2025 plus the 26 added sequences, retaining the 50 removed. Could you confirm which set you evaluated on, and report both trackers on the official 120-sequence set under the official challenge protocol?

Q2. In Table 4's equal-stride control, which rule fills the nine object pointers in SAM3ours — SAM3's native rule or your "recently evicted" rule? The caption specifies only the eviction rule, and Figure 6 shows that swapping the pointer rule alone (with the policy fixed) changes validation performance. If SAM3ours keeps SAM3's pointers, the 1.5-point equal-stride gap (p. 9) still conflates two changes; a three-arm row (SAM3 selection × SAM3 pointers / SAM3 selection × your pointers / SAM3RL) at s = 6 would isolate the policy's contribution.

Q3. Main tables use the best of three seeds chosen on validation (p. 21), while ablation SEMs reach 0.8 (Tables 14–15). Since four of the ten first places lead the runner-up by ≤0.8 — DiDi +0.2, LVOSv2 +0.2, VOT-LT2021 +0.7, LaSOT +0.8 — could you report three-seed means for the main tables?

Q4. DiDi draws 86 of its 180 sequences from the LaSOT test split. There is no training/test overlap with your LaSOT training split, but LaSOT is your only training corpus and your DiDi margin is +0.2. Can you report the DiDi gain split by source — LaSOT-derived (86) versus the rest (94)?

2 comments

Thank you for your careful review, for recognizing that our methodology holds across different backbones, and for pointing out the typo in the VOTS2026 benchmark description. Below we answer each of the weaknesses and questions you raised.

W1 / Q1 — VOTS2026 sequence set and the DAM4SAM3 port

Sequence set. All VOTS2026 numbers in the paper are the official server’s scores on the VOTS2026 split (120 sequences). However, as the official workshop report describes (Kristan et al., 2026), the official toolkit runs a tracker on the union of the VOTS2025 and VOTS2026 sequences (170 sequences, as you noted), and the server scores the two splits separately. Our reported numbers are correctly from the VOTS2026 split (120 sequences), but Table 8 mistakenly listed the statistics of that union instead of the VOTS2026 split. So no score was affected, we just corrected Table 8 in the revised version.

DAM4SAM3 port. We could not find the referenced DAM4SAM3 value 69.3, or a figure with that axis label, in the official workshop report (Kristan et al., 2026). Could you please point us to the figure you mention? The report’s Table 1 instead lists dam4sam_sam3, an independent port by A. Galea, at 59.9, which is 4.6 below the report’s SAM 3 (64.5). Under your relative-gain comparison, that port loses 4.6 against SAM 3, so the official results in fact support our transfer-failure claim. To test the untuned-port explanation, we also tuned DAM4SAM3 on our validation split. This raises its VOTS2026 score from 66.7 to at most 67.8, still 2.2 below SAM3RL (see our response to Reviewer 3HUf, W2/Q2).

SAM 3 baseline. Our SAM 3 baseline (66.4) is indeed higher than the report’s two independent SAM 3 submissions (SAM3a 64.5, SAM3v1 63.1), which themselves differ by 1.4. Differences of this size between papers are common, as we showed in Tables 9 and 10. Therefore, we evaluate all the methods under the same fixed evaluation protocol, specified in Appendix B, to ensure a fair comparison and support reproducibility.

Kristan et al. The Fourth VOTS2026 Challenge Results. VOTS2026 Challenge report, 2026. https://openreview.net/forum?id=zGOan36xpF

Q2 — Gains attributable to the learned eviction

SAM 3 (ours) in the submitted Table 4 indeed used SAM 3’s native object-pointer selection and pointer temporal encoding, not those of SAM3RL. This is because Table 4 was designed as a stride ablation, in which SAM 3 (ours) served as a minimal fix of the official strided SAM 3. The official version collapses because it reselects a shifted bank at every frame, likely a setting SAM 3 never sees in training. The fix keeps the bank unchanged between updates, as our method does, and changes nothing else. To avoid confusion, this row is now called SAM 3 (fix).

We agree that isolating the learned eviction is important. The name SAM (ours) now denotes a new row, which runs SAM 3’s eviction rule inside our MDP (Section 3.1). It shares the stride, the recently evicted pointer rule and the pointer temporal encoding with SAM3RL, so only the eviction policy differs. SAM3RL outperforms SAM 3 (ours) on all seven benchmarks, by 0.9 on average and most on VOT-LT2022 (+2.2) and VOTS2026 (+1.3). Going from SAM 3 (fix) to SAM 3 (ours), which changes only the pointer rule and encoding, adds 0.6 on average. Notably, SAM 3 (ours) alone already outperforms all prior methods on VOTS2026, VOTS2025 and LaSOT, but still falls short of our learned policy. We updated Table 4 and Section 5 accordingly.

ModelsFPSVOTS2026 (Q)VOTS2025 (Q)DiDi (Q)VOT-LT2021 (F)VOT-LT2022 (F)LaSOT (AUC)LaSOT-ext (AUC)
SAM 3115.366.473.471.477.162.875.961.7
SAM3RL114.969.3 (+2.9)75.4 (+2.0)71.7 (+0.3)77.2 (+0.1)65.6 (+2.8)77.1 (+1.2)65.2 (+3.5)
SAM 3615.555.5 (−10.9)66.9 (−6.5)60.5 (−10.9)76.2 (−0.9)61.8 (−1.0)72.0 (−3.9)57.4 (−4.3)
SAM 3 (fix)615.868.1 (+1.7)75.9 (+2.5)71.2 (−0.2)75.7 (−1.4)65.6 (+2.8)77.4 (+1.5)64.8 (+3.1)
SAM 3 (ours)616.168.7 (+2.3)76.6 (+3.2)71.8 (+0.4)77.9 (+0.8)66.1 (+3.3)77.8 (+1.9)64.2 (+2.5)
SAM3RL616.070.0 (+3.6)76.9 (+3.5)72.0 (+0.6)79.0 (+1.9)68.3 (+5.5)78.4 (+2.5)64.7 (+3.0)

W2 / Q3 — Three-seed means for the main tables

Our main tables follow standard practice in tracking, including SAM 2, SAM 3 and the VOTS challenge reports, and compare the benchmark scores of a single released model. For SAM3RL, this is the tracker we release, selected on our validation split without using any test benchmark. The three-seed SEM in Tables 14 and 15 serves a different purpose. It describes the variance of our training method, and we report it for the ablations so that design choices are compared fairly. It shows how much retraining could change a score, but it is not an error bar on comparisons between the released tracker and other methods.

Q4 — DiDi gain split by source

Here we attach the requested gains on DiDi by source (LaSOT-derived vs. the rest). As with most other hand-crafted rules, the SAM3RL gain is concentrated on the 86 LaSOT-derived sequences (+3.2).

SubsetnSAM 3SAMURAI3SAM3LongDAM4SAM3SAMITE3HiM3SAMSAM3RL
DiDi (all)18071.471.8 (+0.4)71.7 (+0.3)71.8 (+0.4)71.8 (+0.4)70.8 (−0.6)72.0 (+0.6)
LaSOT-derived8672.173.0 (+0.9)72.3 (+0.2)74.6 (+2.5)74.1 (+2.0)73.6 (+1.5)75.3 (+3.2)
Other sources9470.870.6 (−0.2)71.2 (+0.4)69.2 (−1.6)69.7 (−1.1)68.2 (−2.6)69.0 (−1.8)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

We thank all three reviewers for their careful reviews and for recognizing the many strengths of our work. We also thank them for the concerns they raised, which we have done our best to address. Below we summarize the main concerns together with the changes we made (the full results are in our individual responses).

Gains attributable to the learned eviction (ukv8, 3HUf, LNGU)

The original SAM 3_ours row in Table 4 did not isolate the contribution of the learned eviction, because it also differed from SAM3RL in the object-pointer selection and the pointer temporal encoding. The table was designed as a stride ablation, but we agreed that isolating the effect of learning is important. We therefore split this row in two: SAM 3 (fix) now serves the original stride ablation, and the new SAM 3 (ours) differs from SAM3RL only in the eviction policy. The new experiment showed that SAM3RL outperforms SAM 3 (ours) on all seven benchmarks, most clearly on VOT-LT2022 and VOTS2026, confirming that consistent gains indeed come from the learned eviction itself.

DAM4SAM3 and SAMITE3 hyperparameter tuning (3HUf)

The reviewer asked for a limited sensitivity study to establish whether the weaker SAM 3 results of these methods reflect their design or their untuned hyperparameters. We added Appendix F, which repeats the sensitivity studies of both original papers on SAM 3, with an additional search over parameter combinations for DAM4SAM3. Tuning brought no improvement for SAMITE3 and only a marginal one for DAM4SAM3, leaving the average gap to SAM3RL essentially unchanged. This indicates that the transfer failure mainly reflects the design of these rules rather than their untuned hyperparameters.

Minor fixes. We corrected Table 8, which listed the statistics of VOTS2025 and VOTS2026 combined instead of VOTS2026 alone (noticed by Reviewer ukv8, but no reported score was affected). We also expanded the explanation of several design choices, such as the full memory bank and foreground pooling, as requested by Reviewer LNGU.

Discussion outcome. We responded to every question the reviewers raised. Reviewer LNGU found the revision clearer and maintained their positive assessment. Reviewers ukv8 and 3HUf have not replied, so their scores stay as they were before our responses and the new experiments. We are happy to make any further changes they request.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.