acceptodds
Under review as a conference paper at ICLR 2027

FastSAM3:Accelerating SAM3 with Lazy Detection and Tracking

Abstract

SAM3 achieves robust performance in video referring segmentation, benefiting from the collaboration and mutual disambiguation between its detector and tracker. However, we observe that the dense per-frame computation of this dual-branch design introduces substantial temporal redundancy, resulting in high computational cost and inference latency: (1) Redundant detection execution. The detector is activated at every frame by default. However, in scenarios such as stable scenes, absence of newly appearing objects, or minimal visual changes, the semantic information becomes largely redundant. (2) Redundant tracking memory. The tracker establishes object associations via cross-attention over a large memory sequence. However, during phases with limited dynamics and distractions, the contribution of most memory frames is marginal, thus involving all of them in computation leads to unnecessary resource consumption. To address these issues, we propose FastSAM3, which introduces lazy detection and tracking to address temporal redundancy and accelerate inference. Specifically, for the detector, we propose Entrant-Predictive Gating, which leverages tracking signal and early-layer responses to assess the likelihood of new object emergence, thereby selectively trigger detection on demand. For the tracker, we propose Divergence-Aware Gating, which switches between sparse memory and full memory based on tracking divergence, performing full computation only when necessary. Without any training, FastSAM3 achieves adaptive computational budget allocation, delivering 1.77 end-to-end speedup with only 0.6 performance drop on the SA-V test set, where lazy detection and lazy tracking provide 2.17 and 1.57 speedup, respectively.

Primary area:
optimization
Keywords:
Segment Anything Model, Adaptive Inference, Video Object Tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KWphOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents FastSAM3, a faster version of SAM3 that doesn’t require any retraining. The original SAM3 applies a detector and tracker to each frame in a video. The detector identifies and segments all relevant objects in the scene, and the tracker re-identifies objects that were detected in previous frames, based on a memory. The paper observes that this process uses unnecessary computation: (i) the object detector is applied to each frame while it only needs to be applied when new objects appear in the scene or the tracker makes an error, and (ii) the tracker uses a memory of many previous frames, whereas for many frames it is sufficient to have only a few reference frames in memory. FastSAM3 leverages this to speed up the model: (i) it only applies the detector when early-layer features suggest that a new object appears or if the tracker’s prediction does not match with the objects in the image, and (ii) its tracker uses a sparse memory with only a few frames by default, and only uses the complete memory when the sparse-memory tracker’s output masks are inconsistent with earlier predictions. With experiments on the SA-Co/VEval benchmark, FastSAM3 is shown to be 1.77x faster than the original SAM3, while only causing a minor drop in segmentation performance.

Critical strengths
  1. The paper presents a well-designed and well-motivated method. It identifies two key inefficiencies in SAM3 and proposes simple but effective improvements for them, resulting in FastSAM3. The fact that these inefficiencies can be solved without significantly harming accuracy is a new finding and contribution that could inspire future follow-up work.

    1. The first identified inefficiency is that the detector is unnecessarily applied to each video frame. As long as no new objects enter the scene and the tracker correctly re-identifies all previously detected objects, it is not necessary for the detector to be applied, as the detector only serves to identify objects that the tracker does not (re-)identify. The paper leverages this redundancy by applying ‘lazy detection’, where the detector is only applied in full when its early-layer features suggest that objects are present at image locations where the tracker does not detect any object. This way, the detector is only applied fully when it is necessary to complement or correct the tracker’s output, reducing computational cost while keeping segmentation performance high.
    2. The second identified inefficiency is that the tracker uses a relatively large memory, with object information from many previous frames, to re-identify new objects. The paper observes that the model only requires such a large memory in case of complex dynamics or motion between frames, which only happens rarely. Therefore, the paper uses a ‘lazy tracker’ by default, which uses a memory with fewer frames, and only uses the tracker with the large memory when the lazy tracker produces outputs that are divergent or inconsistent with previous-frame outputs. By using the less-expensive lazy tracker on most frames and only using the large-memory tracker when necessary, computational cost is reduced while still being able to use the large-memory tracker when necessary to prevent tracking errors.
  2. The method’s effectiveness is adequately demonstrated through experiments. On the SA-Co/VEval benchmark, which is SAM3’s main evaluation benchmark for video-based concept segmentation, FastSAM3 achieves a 1.77x speedup compared to the original SAM3, while only obtaining slightly worse segmentation performance (e.g., 49.8 to 49.5 and 29.1 to 28.0 cgF1\mathrm{cgF}_1 on the YT-Temporal-1B val and SA-V val subsets, respectively). The efficiency improvement at minimal accuracy drop makes FastSAM3 more practically useful for applications than SAM3.

Critical weaknesses

The paper does not have critical weaknesses that should cause it to be rejected. However, there are some smaller points for improvement, which are listed in the ‘Questions for Authors’ section. Considering both the strengths and these smaller points for improvement, I initially give a ‘Weak Accept’ rating. If the authors adequately address the points for improvement, I am willing to upgrade my rating to ‘Clear Accept’.

Questions for authors

There are several smaller points for improvement that could be addressed to further strengthen the paper. If the authors adequately address these points, I’m willing to upgrade my rating.

  1. While the paper is generally clearly written, it contains a substantial number of typos and grammatical errors. The paper’s readability would be improved if these were fixed. Some (non-exhaustive) examples:

    1. L025: “thereby selectively trigger” => “thereby selectively triggering”
    2. L053: “it provide” => “it provides”
    3. L095: “that leverage” => “that leverages”
    4. L203: “2-th” => “2nd”
    5. L206: “where memory feature fusing historical image features” is grammatically incorrect
    6. L225: “can partitions” => “can partition” or “partitions”
    7. L229: “Then,We” => “Then, we”
    8. L286: “overleap” => “overlap”
    9. L263-L264: “which adopt ... and switch” => “which adopts and switches”
    10. L325 & L342: “more metric” => “more metrics”
  2. In Fig. 1, it is not clear how the ‘detection score’ and ‘tracking score’ are computed. Is this detection score the same as the ‘entrant score’ (Eq. 11), and is the tracking score the same as the divergence score (L304)? The paper does not explicitly explain this. As the low values for these scores across video frames are used to demonstrate the detector’s and tracker’s redundancy and to motivate the method in the introduction (L052-L056), it is crucial that the meaning/definition of these scores is clearly specified when they are referred to. The fact that this is currently not done limits the understandability of the paper’s motivation.

  3. In the ‘Memory Bank in Tracker’ paragraph (L205-L215), the paper uses Mt\mathcal{M}_ t to denote the tracker’s memory (L207, Eq. 6). However, earlier in the subsection, Mt\mathcal{M}_ t was used to denote the output of the tracker (Eq. 2), and Ht\mathcal{H}_ t was used to denote the memory features (L187). This is inconsistent and confusing, and should be fixed.

  4. As shown in Tab. 4, the benefit of the temporal divergence score and the identity divergence score is negligible when also using the prediction divergence score to determine whether the tracker should use the sparse or dense memory. In L279-L298, the paper presents the two former divergence scores as being crucial, but in reality using only the prediction divergence appears to be sufficient. The paper should motivate why the temporal divergence score and the identity divergence score are still relevant and necessary, and why it would not be better to just use the prediction divergence score only, potentially yielding an even greater speedup because the other two divergence scores no longer need to be calculated.

  5. From Tab. 1 (L327) and L377-L378, it is not fully clear whether the overall 1.77x speedup truly reflects the speedup of the end-to-end model, from input to output, including the backbone, or whether some components are excluded like for the component-specific speedups of 2.17x and 1.57x. The paper should more clearly explain the measurement setup and motivate this choice.


Note after rebuttal: the authors have answered my questions and addressed my minor concerns, so I upgrade my rating from 'weak accept' to 'clear accept'

6 comments
AuthorsOpenReview ↗

Response to Reviewer KWph (1/2)

We sincerely appreciate your positive assessment and constructive suggestions, and are encouraged by your recognition of our work. We have carefully addressed the raised questions and revised the manuscript accordingly. All changes are highlighted in blue.

Q1: While the paper is generally clearly written, it contains a substantial number of typos and grammatical errors. The paper’s readability would be improved if these were fixed.

R1: Corrections to typos and grammar. [Q1]

Thank you for your careful review. We have thoroughly proofread the manuscript and corrected the typographical and grammatical errors to improve its readability and overall presentation.

Q2: In Fig. 1, it is not clear how the ‘detection score’ and ‘tracking score’ are computed. Is this detection score the same as the ‘entrant score’ (Eq. 11), and is the tracking score the same as the divergence score (L304)? The paper does not explicitly explain this. As the low values for these scores across video frames are used to demonstrate the detector’s and tracker’s redundancy and to motivate the method in the introduction (L052-L056), it is crucial that the meaning/definition of these scores is clearly specified when they are referred to. The fact that this is currently not done limits the understandability of the paper’s motivation.

R2: Definitions of the scores in Figure 1. [Q2]

Thank you for your valuable question. The Detection Score and Tracking Score in Fig. 1 are distinct from the entrant score and divergence scores used in our method. More specifically, they quantify the actual contributions of detection and dense memory to the model outputs at each frame, respectively, under the same historical state. Their computation is detailed below.

1. Detection Contribution Score. We quantify the contribution of detection by comparing the outputs with and without executing the detection branch under the same historical state. Three cases are considered: (1) if no objects are detected, the contribution is 00; (2) if no new objects are detected, the contribution is measured by the maximum confidence-weighted mask discrepancy between the two outputs; and (3) if new objects are detected, the contribution is measured by the sum of their confidence scores. Formally,

\begin{cases} 0, & A_t=B_t=\varnothing,\\\\[2mm] \displaystyle\max_{i\in A_t} s_{t,i}\left[1-\operatorname{IoU} \left(M_{t,i}^{D},M_{t,i}^{\neg D}\right)\right], & A_t=B_t\neq\varnothing,\\\\[3mm] \displaystyle\sum_{i\in A_t\setminus B_t}s_{t,i}, & A_t\neq B_t \land A_t\setminus B_t\neq\varnothing, \end{cases}$$ where $A_t$ and $B_t$ denote the object sets obtained with and without executing the detection branch, respectively. $M_{t,i}^{D}$ and $M_{t,i}^{\neg D}$ represent the corresponding object masks, and $s_{t,i}$ denotes the confidence score of object $i$ in the full detection output. **2. Full-Memory Contribution Score.** We quantify the contribution of full memory by comparing the tracking outputs obtained using full and sparse memory under the same historical state. The score is defined as the maximum IoU discrepancy between the resulting tracking masks: $$C_{\mathrm{FM}}(t)= \max_i\left[ 1-\operatorname{IoU} (M_{t,i}^{F},M_{t,i}^{S}) \right].$$ where $M_{t,i}^{F}$ and $M_{t,i}^{S}$ denote the binary tracking masks of object $i$ obtained using full and simplified memory, respectively, and $i$ indexes the tracked objects. When both masks are empty, their IoU is defined as $1$. For clarity, we have added the definitions and detailed computation procedures of both contribution scores in the Appendix A.5. The temporally sparse high values indicate that detection and dense memory contribute little in most frames, motivating adaptive execution. >Q3: In the ‘Memory Bank in Tracker’ paragraph (L205-L215), the paper uses $\mathcal{M}_t$ to denote the tracker’s memory (L207, Eq. 6). However, earlier in the subsection, $\mathcal{M}_t$ was used to denote the output of the tracker (Eq. 2), and $\mathcal{H}_t$ was used to denote the memory features (L187). This is inconsistent and confusing, and should be fixed. > **R3: Consistent notation for tracker outputs and memory features. [Q3]** Thank you for pointing out this notation inconsistency. We have revised the mathematical notation in the corresponding subsection, reserving $\mathcal{M}_t$ for the tracker outputs and consistently using $\mathcal{H}_t$ for the memory features. We have also checked the related equations and descriptions to ensure consistency throughout the paper.
AuthorsOpenReview ↗

Response to Reviewer KWph (2/2)

Q4: As shown in Tab. 4, the benefit of the temporal divergence score and the identity divergence score is negligible when also using the prediction divergence score to determine whether the tracker should use the sparse or dense memory. In L279-L298, the paper presents the two former divergence scores as being crucial, but in reality using only the prediction divergence appears to be sufficient. The paper should motivate why the temporal divergence score and the identity divergence score are still relevant and necessary, and why it would not be better to just use the prediction divergence score only, potentially yielding an even greater speedup because the other two divergence scores no longer need to be calculated.

R4: Roles of temporal and identity divergence. [Q4]

Thank you for your valuable question. While prediction divergence leverages model's distractor awareness to identify disagreement among candidate masks, temporal and identity divergence capture motion inconsistency and cross-object identity conflicts, respectively.

To further examine their contributions, we extend the ablation study to other dataset splits. As shown in the following table, incorporating all three signals generally improves performance over prediction-only gating, with more noticeable gains on SA-V. Although the improvements are modest and vary across datasets, these results suggest that the additional signals provide complementary benefits.

DatasetGating SignalscgF1pmF1pHOTApDetA
YT1B_valPrediction only49.155.970.159.9
All49.456.370.159.9
YT1B_testPrediction only49.856.269.259.5
All49.856.269.359.5
SA-V_valPrediction only28.443.460.344.4
All28.943.960.644.6
SA-V_testPrediction only29.342.657.640.5
All29.742.957.640.5
SmartGlasses_valPrediction only32.142.259.044.9
All32.342.359.345.4
SmartGlasses_testPrediction only34.744.462.549.1
All35.345.162.649.1

We also analyze the triggering statistics of individual signals across three test sets. Prediction divergence and low confidence are the main triggers. Nevertheless, exclusive triggers from temporal and ID divergence account for 8.5%–18.9% of all dense-memory triggers, covering cases missed by the other conditions. Removing these signals reduces dense computation, but would also eliminate these independent checks, increasing the risk of tracking failures.

DatasetTrigger ConditionTrigger rate (%)Exclusive trigger rate (%)Trigger share (%)
YT1B_testPrediction divergence6.721.8969.81
ID divergence0.830.138.63
Temporal divergence1.971.0020.47
Low confidence6.421.5466.69
Any9.63--
SA-V_testPrediction divergence3.941.2062.34
ID divergence0.270.064.35
Temporal divergence0.820.4812.93
Low confidence4.501.7371.23
Any6.32--
SmartGlasses_testPrediction divergence2.610.6853.01
ID divergence0.280.085.59
Temporal divergence1.120.8522.67
Low confidence3.261.2966.13
Any4.93--

Q5: From Tab. 1 (L327) and L377-L378, it is not fully clear whether the overall 1.77x speedup truly reflects the speedup of the end-to-end model, from input to output, including the backbone, or whether some components are excluded like for the component-specific speedups of 2.17x and 1.57x. The paper should more clearly explain the measurement setup and motivate this choice.

R5: Clarification of speedup measurements and end-to-end latency. [Q5]

Thank you for pointing out this ambiguity. The 1.77× speedup refers to the combined detector and tracker branches, excluding backbone computation. We use this branch-level measurement to isolate the computational savings of EPG and DAG, which specifically optimize these two branches.

Table 6 in Appendix A.2 of the original submission provides a detailed latency breakdown, reproduced below. Including the backbone, full-pipeline latency decreases from 317 ms to 226 ms, corresponding to a 1.40× speedup. We have revised the manuscript to clearly distinguish the branch-level speedup (1.77×) from the full-pipeline speedup (1.40×), and corrected the corresponding descriptions to avoid ambiguity.

Time (ms)BackboneDetectorTracker-MemoryTracker-DecoderOthersTotalDetector+Tracker
SAM398100772319317200
LD-EPG9546 (2.17x)721719249135
LT_DAG9510149 (1.57x)2719291177
Fast-SAM39442492219226 (1.40x)113 (1.77x)

I thank the authors for their detailed response to my review.

With the response, the authors have partly addressed my concerns and answered my questions:

  • Q1: The authors have fixed the textual errors that I included in my review as examples. However, multiple other errors still exist in the paper. Therefore, I do not consider this concern fully addressed. I would recommend the authors to do a more thorough proofread and fix the remaining errors. Some further examples (again not an exhaustive list of all errors):
    • L064: "where detector's false positives" => "where the detector's false positives"
    • L096: "to adaptively determines" => "to adaptively determine"
    • L109: "For detector" => "For the detector"
    • L300: "become low" => "becomes low"
  • Q2: The authors have explained how the detection score and tracking score used in Fig. 1 are computed, and have added this explanation to the appendix. However, the explanation states that the detection score is the sum of the confidence scores of all new objects that are detected (which can be larger than 1.0), while the maximum value for the detection score in Fig. 1 is 1.0. This discrepancy is confusing and needs to be explained. Therefore, I also consider this question only partially answered.
  • Q3: The authors have adequately addressed my concern about L205-L215 by fixing the notation.
  • Q4: In their response, the authors have provided additional experiments to show that, in some experimental settings, the temporal divergence score and identity divergence score do improve the performance slightly more than in Tab. 4, compared to using only the prediction divergence score. While the performance improvements are still minimal, I can see that there is some benefit to the temporal and identity divergence scores. Therefore, I consider this concern addressed. However, I would encourage the authors to include the provided experimental results also in the appendix of the paper, so other readers also have access to this information.
  • Q5: The authors have adequately answered my question by pointing out that the 1.77x speedup reflects only the speedup of the detector and tracker, and that that the overall model speedup is 1.40x. The authors have attempted to also clarify this in the paper by using the term 'dual-branch speedup' to indicate the 1.77x speedup. However, I believe it may not be immediately clear to the reader what 'dual-branch speedup' means. Also, I believe it would be relevant for the reader to be informed about the 1.40x overall model speedup in the main paper already, and not only in the appendix. Therefore, I would encourage the authors to better clarify in the paper what 'dual-branch speedup' means, and also mention the 1.40x overall model speedup in the main paper.

I've also carefully read the other reviews and the authors' responses to them, and I believe the comments in these reviews have mostly been properly addressed. I just have one follow-up question:

  • In response R5 to reviewer YZog, the authors state that DAG can identify occlusion-induced tracking divergence. Could the authors explain how DAG achieves this, e.g., how it identifies a case where the tracker fails to re-identify an object that was occluded for a relatively large number of frames?

Since my concerns and questions have been addressed only partly, I keep my 'weak accept' rating unchanged.

Reviewer_7V2mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FastSAM3 is a training-free inference framework for reducing redundant detection and memory computation in SAM3 video concept segmentation. It uses early detector responses and tracking signals to decide whether to complete detection, and switches from sparse to full tracking memory when divergence or low confidence indicates unreliable predictions. Evaluations across multiple video benchmarks report segmentation and tracking quality close to the original SAM3, while a separate timing example demonstrates full-pipeline acceleration.

Critical strengths
  1. The method addresses temporal redundancy in detection and tracking through a well-motivated design that requires no additional training.
  2. Evaluations across multiple video datasets and component ablations support the quality preservation achieved by the proposed gating mechanisms.
Critical weaknesses
  1. The methodological contribution could be positioned more precisely relative to prior adaptive tracking and memory methods. DAM4SAM [1] already uses mask disagreement to identify distractors, while Efficient-SAM2 [2] exploits sparse memory retrieval. These connections are acknowledged; distinguishing the inherited signals from the new gating and coordination mechanisms would make the contribution clearer.

  2. The architectural scope of the approach could be clarified further. Evaluation is limited to SAM3, and the gates depend on early fusion-layer responses, multiple mask predictions, and a particular memory structure. Transferability to other model configurations has not been established. A discussion of reusable components and architecture-specific adaptations would help readers assess its broader applicability.

[1] Videnovic et al., A Distractor-Aware Memory for Visual Object Tracking with SAM2, 2025.
[2] Zhang et al., Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval, 2026.

Questions for authors
  1. Could you identify which aspects of the gating mechanisms are adopted, adapted, or newly introduced relative to DAM4SAM [1] and Efficient-SAM2 [2]? In particular, explaining the additional insight behind using these signals to control detector execution and memory fallback would help distinguish the contribution from existing reliability estimation and memory sparsification techniques. A concise mechanism-level comparison would be helpful.

  2. Could you outline the assumptions required to apply the approach to another model configuration? It would be useful to explain how the early-exit location, entrant score, and divergence criteria depend on SAM3’s internal representations and outputs. A brief account of what can be reused and what requires adaptation would clarify the current scope and guide future validation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7V2m (1/2)

We sincerely thank you for valuable comments and suggestions. Below, we provide detailed responses to clarify the contributions of FastSAM3 and hope these explanations address your concerns.

W1: The methodological contribution could be positioned more precisely relative to prior adaptive tracking and memory methods. DAM4SAM [1] already uses mask disagreement to identify distractors, while Efficient-SAM2 [2] exploits sparse memory retrieval. These connections are acknowledged; distinguishing the inherited signals from the new gating and coordination mechanisms would make the contribution clearer.

Q1:Could you identify which aspects of the gating mechanisms are adopted, adapted, or newly introduced relative to DAM4SAM [1] and Efficient-SAM2 [2]? In particular, explaining the additional insight behind using these signals to control detector execution and memory fallback would help distinguish the contribution from existing reliability estimation and memory sparsification techniques. A concise mechanism-level comparison would be helpful.

R1: Contributions and distinctions from prior work. [W1, Q1]

Thank you for this valuable suggestion. We therefore revisit the positioning of our contributions and clarify both the insights drawn from previous studies and the new gating mechanisms introduced in FastSAM3.

1. Core contributions of FastSAM3.

FastSAM3 aims to reduce temporal redundancy in both the detection and tracking branches of SAM3 through two complementary gating mechanisms.

  • EPG leverages early-layer attention and tracking information to predict potential new object appearances, allowing unnecessary detector execution to be skipped.
  • DAG combines prediction, temporal, and identity divergence to assess tracking reliability and dynamically switches between sparse and dense memory.

The key contribution lies in designing these two gating mechanisms to translate detection cues and tracking reliability into adaptive computation decisions, enabling demand-driven computation allocation across SAM3's detection and tracking branches.

2. Relationship to prior work.

  • Both DAM4SAM[1] and DAG exploit the model’s inherent distractor awareness. However, DAM4SAM uses this property to organize memory at each time step, whereas DAG uses it as one of reliability signals to determine when to switch from sparse to dense memory across time steps. DAG’s contribution lies in integrating three complementary divergence signals into a temporal computation allocation mechanism, selectively activating dense memory when necessary.
  • Efficient-SAM2 [2] reduces memory-attention computation through token-level sparse memory retrieval (SMR). In contrast, our DAG introduces temporal, frame-level switching between sparse and dense memory based on tracking reliability. These two mechanisms operate on different dimensions: SMR determines which memory tokens to retain, whereas DAG determines when to use sparse or full memory. Consequently, DAG is compatible with different memory sparsification strategies, including SMR and MemPool, as demonstrated in Table 5.

In addition to DAG's tracking optimization, EPG introduces a novel gating mechanism to reduce redundant detector execution by leveraging early-layer attention and tracking information to predict potential new object appearances. We clarify these connections and better distinguish the insights drawn from prior work in the following table:

MethodComponentAttributionRole in FastSAM3Distinction to related work
EPGDetector Execution GatingNewly introducedAdaptively skips unnecessary detector execution.-
Entrant PredictionNewly introducedUses the detector’s early-layer features, combined with tracking foreground-background information, to predict the likelihood of new object emergence.-
DAGSparse-Dense Memory GatingNewly introducedAdaptively switches between sparse and dense memory according to tracking reliability.-
Sparse Memory ModeAlternative designSMF: Adopts frame-level memory sparsification.Efficient-SAM2: adopts token-level memory sparsification.
Prediction DivergenceAdaptedAdapts distractor-awareness to identify unreliable tracking states and trigger dense memory.DAM4SAM: introduce distractor-awareness to organize memory and improve tracking.
ID DivergenceNewly introducedIdentifies identity conflicts among multiple tracked objects.-
Temporal DivergenceNewly introducedIdentifies spatiotemporal inconsistencies in tracking.-
AuthorsOpenReview ↗

Response to Reviewer 7V2m (2/2)

W2: The architectural scope of the approach could be clarified further. Evaluation is limited to SAM3, and the gates depend on early fusion-layer responses, multiple mask predictions, and a particular memory structure. Transferability to other model configurations has not been established. A discussion of reusable components and architecture-specific adaptations would help readers assess its broader applicability.

Q2: Could you outline the assumptions required to apply the approach to another model configuration? It would be useful to explain how the early-exit location, entrant score, and divergence criteria depend on SAM3’s internal representations and outputs. A brief account of what can be reused and what requires adaptation would clarify the current scope and guide future validation.

R2: Architectural scope and requirements for transferability. [W2, Q2]

Thank you for raising this important point. FastSAM3 is designed for SAM3's complementary dual-branch detection-tracking architecture, and its transferability to other architectures has not yet been experimentally established. However, its core principles can be extended to similar architectures to improve system efficiency. The full framework requires separately executable detection and tracking branches, while EPG and DAG have distinct requirements and may be adapted individually.

In general, such architectures consist of two complementary but independently operating branches: a detector for semantic object recognition and a tracker for maintaining temporal associations across instances. More specifically, the two gating mechanisms have different architectural requirements:

1. Lazy Detection (EPG) predicts potential new objects to determine whether full detector execution is necessary. Its transfer requires:

  • Required inputs: Accessible intermediate detector features and tracking foreground–background references.
  • Early-exit location: In SAM3, early Fusion Encoder attention already highlights relevant objects (Figure 2). For ViT-based detectors with similar characteristics, the gating layer should be selected where target-oriented cues become sufficiently informative.
  • Entrant score: The foreground–background comparison can be reused, while candidate extraction, similarity features, and score thresholds require validation and adaptation to the target detector.

2. Lazy Tracking (DAG) adaptively switches between sparse and dense memory based on tracking reliability. Its transfer requires:

  • Memory mechanism: Configurable memory retrieval supporting both sparse and dense modes.
  • Temporal and identity divergence: Consecutive tracking outputs and identity information, with divergence criteria and thresholds adapted to the target tracker.
  • Prediction divergence: Multiple candidate masks and their confidence scores, as provided by SAM2 and SAM3, with thresholds validated for the target tracker.

The following table summarizes the reusable principles, architectural requirements, and necessary adaptations of each component.

ComponentReusable PrincipleRequirements and Adaptations
EPG: Detection GatingExecute full detection only when needed.Separately executable detection and tracking branches.
EPG: Entrant PredictionIdentify potential untracked objects using detector features and tracking references.Accessible intermediate features with target-oriented cues; validate and adapt the gating layer, candidate extraction, similarity features, and thresholds.
DAG: Memory SwitchingUse sparse memory by default and activate dense memory when tracking is unreliable.Configurable memory retrieval with access to both sparse and dense memory.
DAG: Temporal DivergenceAssess reliability through temporal consistency.Consecutive tracking outputs; adapt consistency measures and thresholds to the target tracker.
DAG: Identity DivergenceDetect potential conflicts among tracked identities.Multi-object predictions and identity information; adapt the conflict criterion and threshold.
DAG: Prediction DivergenceAssess ambiguity through disagreement among candidate masks.Multiple candidate masks and confidence scores, as in SAM2/SAM3; validate thresholds for the target tracker.

Thanks for the responses. I will keep my score unchanged.

Thank you for your positive assessment and careful consideration of our responses. We sincerely appreciate your constructive feedback and valuable suggestions, which have helped us further clarify our contributions and improve the manuscript. We are grateful for your time, thoughtful review, and recognition of our work.

Reviewer_YZogOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FastSAM3, a framework for speed-up of SAM3 model. Authors focus on 2 parts: redundant detection execution and redundant tracking memory. The first one is resolved by leveraging tracking signals and early-layer responses to assess if the new object will appear in a given frame (thereby determining if the detector needs to be run). The second one is resolved by switching between sparse (4 memory slots) and dense (10 memory slots; standard for SAM3) tracking memory, with switch determined by analyzing prediction, temporal, and identity divergence. This leads to 1.4x speedup with neglible decrease in tracking performance.

Critical strengths

Strength 1: Interesting insights on how to increase the speed of SAM3 inference. Authors propose a clean split in lazy tracking on prediction, temporal, and identity divergence analysis.

Critical weaknesses

Weakness 1: Limited novelty compared to previous works. Some modules (including prediction divergence) are based on known components from previous trackers. The lazy detection part is conceptually similar to DAM4SAM, in which one of the core insights is to leverage information about distractors in the background. Lazy tracking includes prediction divergence, inspired by DAM4SAM.

Weakness 2: Evaluation needs more datasets with challenging tracking scenarios, e.g., LaSOT[1] and DiDi[2]. In those cases, the switches between the sparse and dense tracking memory could be more important due to similar objects and occlusions, and it would be interesting to see if the designed modules would be robust to this.

[1] LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking, Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, Haibin Ling; CVPR, 2019

[2] A Distractor-Aware Memory for Visual Object Tracking with SAM2, Jovana Videnovic, Alan Lukezic, Matej Kristan; CVPR, 2025

Questions for authors

Question 1: Can you evaluate performance and speed for two simple baselines? In the first one, the tracking memory holds the two most recent frames and the two highest confidence conditioning frames. For the second experiment, it would use the two most recent frames and the two most recent conditioning frames.

Question 2: Can you provide qualitative visualizations that would show that entrants-predictive gating is effective? One simple visualization would be to visualize the similarity map on top of the given frame.

Question 3: Can you extend the evaluation on more datasets? An interesting experiment would be running SAM3 with lazy tracking on dataset with more similar objects and occlusions, e.g., DiDi or LaSOT.

Question 4: How is the occlusion of the object handled in FastSAM3? Does it influence lazy detection or lazy tracking parts? Can you evaluate the tracking performance and speedup during different levels of occlusion (none, partial, full)?

Question 5: For each part of divergence-aware gating for lazy tracking, can you analyze how frequently it triggers a switch from sparse to dense memory?

7 comments
AuthorsOpenReview ↗

Response to Reviewer YZog (1/4)

We sincerely thank you for valuable comments and constructive suggestions. We have conducted additional experiments and provide point-by-point responses below. We hope these results and explanations clarify our contributions and address your concerns.

W1: Limited novelty compared to previous works. Some modules (including prediction divergence) are based on known components from previous trackers. The lazy detection part is conceptually similar to DAM4SAM, in which one of the core insights is to leverage information about distractors in the background. Lazy tracking includes prediction divergence, inspired by DAM4SAM.

R1: Contributions and distinctions from prior work. [W1]

Thank you for this valuable feedback. We would like to clarify that FastSAM3 and DAM4SAM address different problems. DAM4SAM focuses on improving tracking robustness through distractor-aware memory management, whereas FastSAM3 targets temporal computational redundancy in SAM3's detection-tracking pipeline. By exploiting the complementary roles of the detector and tracker, FastSAM3 adaptively allocates computation across video frames. Specifically:

  • Lazy Detection (EPG): EPG predicts the likelihood of new object emergence at an early layer of the detector and determines whether to proceed with full detection. Its core idea is to leverage detector's early-layer features, combined with tracker's foreground-background information, to identify potential untracked objects (entrants), thereby guiding detection execution. Importantly, the tracker provides foreground-background mask to distinguish entrants, rather than distractor cues to identify distractors. Thus, EPG performs entrant prediction and detector execution gating, which DAM4SAM does not address.

  • Lazy Tracking (DAG): DAG adaptively switches between sparse and dense memory according to tracking reliability, using three divergence signals (prediction, ID, and temporal). Both DAM4SAM and DAG's prediction divergence exploit the model’s inherent distractor awareness. However, DAM4SAM uses this property to organize memory at each time step, whereas DAG uses it to determine when to switch from sparse to dense memory across time steps. DAG’s contribution lies in integrating complementary reliability signals into a temporal computation allocation mechanism, selectively activating dense memory when necessary.

To clarify FastSAM3’s contributions and its connections to related work, we summarize the corresponding components below:

MethodComponentRole in FastSAM3Distinction to related work
EPGDetector Execution GatingAdaptively skips unnecessary detector execution.-
Entrant PredictionUses the detector’s early-layer features, combined with tracking foreground-background information, to predict the likelihood of new object emergence.-
DAGSparse-Dense Memory GatingAdaptively switches between sparse and dense memory according to tracking reliability.-
Sparse Memory ModeAdopts frame-level memory sparsification.Efficient-SAM2: adopts token-level memory sparsification.
Prediction DivergenceUses distractor-awareness to identify unreliable tracking states and trigger dense memory.DAM4SAM: uses distractor-awareness to organize memory and improve tracking.
ID DivergenceIdentifies identity conflicts among multiple tracked objects.-
Temporal DivergenceIdentifies spatiotemporal inconsistencies in tracking.-
AuthorsOpenReview ↗

Response to Reviewer YZog (2/4)

W2: Evaluation needs more datasets with challenging tracking scenarios, e.g., LaSOT and DiDi. In those cases, the switches between the sparse and dense tracking memory could be more important due to similar objects and occlusions, and it would be interesting to see if the designed modules would be robust to this.

Q3:Can you extend the evaluation on more datasets? An interesting experiment would be running SAM3 with lazy tracking on dataset with more similar objects and occlusions, e.g., DiDi or LaSOT.

R2: Extend evaluation on challenging tracking benchmarks. [W2, Q3]

Thank you for your valuable suggestion. The original submission has reported performance on GMOT and AnimalTrack datasets in Appendix A.7, addressing crowded scenes with visually similar objects. Following the your suggestion, we have extended the evaluation to LaSOT and DiDi to further assess the robustness of adaptive memory switching under similar-object interference and occlusion. We follow single-object tracking protocols, using SAM3’s tracker without activating the detection branch.

On DiDi, DAG outperforms both fixed sparse memory and the dense baseline in Quality and Robustness. On LaSOT, it improves AUC by 4.89 points over fixed sparse memory, approaching the dense baseline. These results support the effectiveness of selectively activating dense memory to preserve tracking reliability in challenging scenarios.

DiDi benchmark results:

MethodSparse ratio ↑\uparrowQuality ↑\uparrowAccuracy ↑\uparrowRobustness ↑\uparrowNRE ↓\downarrowDRE ↓\downarrowADQ ↑\uparrow
SAM3 (dense)072.3275.1695.081.823.1066.10
SMF (sparse)170.9274.8593.623.063.3264.70
LT-DAG (ours)0.97472.8475.3895.571.882.5469.62

LaSOT benchmark results:

MethodSparse ratio ↑\uparrowAUC ↑\uparrowP@20 ↑\uparrowNP@ 0.2 ↑\uparrow
SAM3 (dense)077.1281.6784.72
SMF (sparse)171.8575.3378.20
LT-DAG (ours)0.98476.7481.2984.03

Q1: Can you evaluate performance and speed for two simple baselines? In the first one, the tracking memory holds the two most recent frames and the two highest confidence conditioning frames. For the second experiment, it would use the two most recent frames and the two most recent conditioning frames.

R3: Comparison with other memory selection baselines. [Q1]

Thank you for your thoughtful suggestion. Following your suggestion, we evaluated R2+CC2, retaining the two most recent frames and the two highest-confidence conditioning frames, and R2+RC2, retaining the two most recent frames and the two most recent conditioning frames. Both use the same memory length as SMF and achieve the same 1.64× component-level speedup, compared with 1.57× for LT-DAG.

Both baselines underperform LT-DAG across all evaluation splits. These results support the importance of adaptive sparse-dense memory switching, rather than a better fixed memory selection strategy, in achieving DAG’s performance gains.

DatasetMethodcgF1pmF1pHOTApDetA
YT-Temporal-1B testSMF48.554.868.559.1
R2+CC248.254.568.459.1
R2+RC248.154.468.259.0
LT-DAG (ours)49.856.269.359.5
YT-Temporal-1B valSMF48.455.169.359.1
R2+CC247.854.569.059.1
R2+RC247.854.569.159.2
LT-DAG (ours)49.456.370.159.9
SmartGlasses testSMF33.943.461.748.4
R2+CC233.442.861.748.6
R2+RC233.142.461.448.5
LT-DAG (ours)35.345.162.649.1
SA-V testSMF29.242.157.340.3
R2+CC229.442.757.440.3
R2+RC229.042.157.240.3
LT-DAG (ours)29.742.957.640.5
AuthorsOpenReview ↗

Response to Reviewer YZog (3/4)

Q2: Can you provide qualitative visualizations that would show that entrants-predictive gating is effective? One simple visualization would be to visualize the similarity map on top of the given frame.

R4: Qualitative visualization of Entrants-Predictive Gating. [Q2]

Thanks for your useful suggestion. EPG uses the tracker’s output mask to separate tracked regions from the backgroundand early-layer attention to select semantically relevant candidates. Feature similarity then helps filter out noise and determines the final entrant score SES_E.

Figures 2 and 9 in the original submission show attention’s preference for target-relevant regions. We added qualitative visualizations in the Appendix A.7, to illustrate how EPG uses this property for entrant prediction. These visualizations show attention heatmaps, feature similarity heatmaps, and predicted entrants, with contours marking tracked objects. Attention highlights semantically relevant entrant candidates, while feature similarity helps filter out noise. As shown in the figures, EPG effectively identifies partially visible incoming objects at the image boundary (red circles), demonstrating its ability to predict entrants appearance.

Q4: How is the occlusion of the object handled in FastSAM3? Does it influence lazy detection or lazy tracking parts? Can you evaluate the tracking performance and speedup during different levels of occlusion (none, partial, full)?

R5: Occlusion handling and robustness evaluation. [Q4]

Thank you for your valuable question. Occlusion can compromise sparse-memory tracking, particularly when objects reappear. FastSAM3 addresses this challenge through complementary mechanisms: EPG triggers detection to recover missed objects, while DAG identifies unreliable tracking states and activates dense memory to provide richer historical information.

  • For Lazy Detection, EPG identifies tracker-missed reappearing objects as entrants and triggers detection, allowing SAM3’s native temporal disambiguation mechanism to correct these tracking failure. For example, in Figure 4 (third row, third frame), the missed reappearing object yields a high SES_E, triggering detection and enabling the the correction.

  • For Lazy Tracking, DAG identify occlusion-induced tracking divergence and trigger a switch to dense memory to support re-tracking. For example, in Figure 4 (third row, fourth and fifth frames), a triggered switch to dense memory helps recover lost objects.

Following your suggestion, we further evaluated tracking performance and efficiency on LaSOT subsets with different occlusion attributes: no occlusion (None), partial occlusion (POC), and full occlusion (FOC), with overlap between POC and FOC. The results show that DAG achieves near-dense tracking performance on both occlusion subsets, substantially outperforming fixed sparse memory while activating dense memory more frequently, supporting FastSAM3’s effectiveness in handling occlusion.

MethodOverall (280)None (76)POC (187)FOC (118)
AUCSparsityAUCSparsityAUCSparsityAUCSparsity
SAM3 (dense)77.12082.85075.83070.580
SMF (sparse)71.85177.53170.40165.861
FastSAM3 (ours)76.740.98481.930.99775.810.97970.280.973

To evaluate DAG’s speedup under different occlusion conditions, we selected representative videos from each of the None, POC, and FOC subsets, with sparsity close to the corresponding subset average. The table below reports memory-attention GPU latency and speedup relative to dense-memory SAM3.

OcclusionMethodms/frameSparsitySpeedup
NoneSAM3 (dense)7.3401×
SMF (sparse)3.9911.841×
FastSAM3 (ours)4.040.9971.816×
POCSAM3 (dense)7.2901×
SMF (sparse)3.9311.857×
FastSAM3 (ours)4.010.9901.820×
FOCSAM3 (dense)7.1301×
SMF (sparse)3.9311.815×
FastSAM3 (ours)4.050.9681.760×
OverallSAM3 (dense)7.2501×
SMF (sparse)3.9511.836×
FastSAM3 (ours)4.040.9841.797×

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary of Submission 24857

Dear Area Chair,

Thank you for your time and consideration. We sincerely thank all reviewers for their constructive suggestions and valuable insights. Below is a summary of our response.

First, we are encouraged by the reviewers’ recognition of FastSAM3’s motivation, gating design, and speed–accuracy trade-off. KWph noted that FastSAM3 identifies “two key inefficiencies in SAM3” and proposes “simple but effective improvements.” 7V2m and KWph highlighted the motivation behind lazy detection and lazy tracking, with 7V2m also noting the training-free design. YZog recognized the “clean split” into prediction, temporal, and identity divergence for adaptive memory switching. 7V2m also noted that evaluations across multiple video datasets and component ablations support the preservation of segmentation quality. KWph highlighted the practical value of the 1.77× speedup for the detection and tracking branches. In particular, KWph characterized FastSAM3’s reduction of both sources of redundancy with little loss in accuracy as “a new finding and contribution that could inspire future follow-up work.”

Second, the reviewers also offered several constructive suggestions. In response to the reviewers’ feedback, we clarified our contributions, conducted additional experiments, and revised the manuscript, as detailed below.

  • We clarified FastSAM3’s contributions of entrant-aware detection and adaptive memory switching, highlighting their technical distinctions from DAM4SAM and Efficient-SAM2 (YZog W1; 7V2m W1,Q1).
  • We extended evaluation on two additional challenging tracking benchmarks, and analysis the DAG's performance under different occlusion conditions (YZog W2,Q3,Q4). We also compared DAG with two additional fixed memory selection baselines (YZog Q1). Together, these experiments confirm DAG’s robustness and show that switching to dense memory when needed improves tracking performance with little added computation.
  • We analyzed the trigger frequency of DAG's each gating signal and conducted ablation studies on more datasets to evaluate the contributions of identity and temporal divergence. These analyses confirmed their importance as complementary signals (YZog Q5, KWph Q4).
  • We discussed the transferability of FastSAM3’s gating mechanisms and outlined requirements for adapting them to other architectures (7V2m W2,Q2).
  • We detailed how the detection and full-memory contribution scores in Figure 1 are computed and visualized (KWph Q2).
  • We revised the manuscript to avoid confusion and improve readability, reproducibility, and overall presentation.
    • Explicitly reported the 1.40× end-to-end speedup as the primary result, while clearly distinguishing it from the 1.77× component-level speedup (KWph Q5).
    • Corrected grammatical and spelling errors (KWph Q1), unified the mathematical notation (KWph Q3).
    • Added more quantitative analysis of EPG with supporting visualizations (YZog Q2).

After our initial responses, reviewer YZog still had concerns about “the novelty of the lazy-tracking mechanism and the whole pipeline accuracy-speed trade-off compared to the two simple baselines.” We addressed these concerns in a follow-up response covering three main points.

  • Novelty and transferability. We further clarified FastSAM3’s novelty, particularly the distinction in scope between DAG and DAM4SAM. We also discussed the broader research implications and transferability of FastSAM3.
  • Complementary gating signals. We clarified how identity and temporal divergence contribute to DAG’s tracking performance by capturing cases that other signals miss.
  • Accuracy–speed trade-off. We explained how DAG’s selective use of dense memory improves tracking robustness over fixed memory selection baselines at a modest computational cost.

Although Reviewer YZog did not provide further feedback, we hope our follow-up responses help address these remaining concerns.

We believe our responses address most of the concerns raised by the reviewers, and we were encouraged by their follow-up feedback. During the discussion, Reviewer 7V2m confirmed their Weak Accept rating, stating, “I will keep my score unchanged.” Reviewer KWph recognized the technical novelty of our work, concluded that our responses “adequately address my questions and the remaining concerns that I had,” and raised the rating from Weak Accept to Clear Accept.


Finally, we again thank all reviewers for their comments and especially thank the ACs for their time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.