- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents FastSAM3, a faster version of SAM3 that doesn’t require any retraining. The original SAM3 applies a detector and tracker to each frame in a video. The detector identifies and segments all relevant objects in the scene, and the tracker re-identifies objects that were detected in previous frames, based on a memory. The paper observes that this process uses unnecessary computation: (i) the object detector is applied to each frame while it only needs to be applied when new objects appear in the scene or the tracker makes an error, and (ii) the tracker uses a memory of many previous frames, whereas for many frames it is sufficient to have only a few reference frames in memory. FastSAM3 leverages this to speed up the model: (i) it only applies the detector when early-layer features suggest that a new object appears or if the tracker’s prediction does not match with the objects in the image, and (ii) its tracker uses a sparse memory with only a few frames by default, and only uses the complete memory when the sparse-memory tracker’s output masks are inconsistent with earlier predictions. With experiments on the SA-Co/VEval benchmark, FastSAM3 is shown to be 1.77x faster than the original SAM3, while only causing a minor drop in segmentation performance.
Critical strengths
-
The paper presents a well-designed and well-motivated method. It identifies two key inefficiencies in SAM3 and proposes simple but effective improvements for them, resulting in FastSAM3. The fact that these inefficiencies can be solved without significantly harming accuracy is a new finding and contribution that could inspire future follow-up work.
- The first identified inefficiency is that the detector is unnecessarily applied to each video frame. As long as no new objects enter the scene and the tracker correctly re-identifies all previously detected objects, it is not necessary for the detector to be applied, as the detector only serves to identify objects that the tracker does not (re-)identify. The paper leverages this redundancy by applying ‘lazy detection’, where the detector is only applied in full when its early-layer features suggest that objects are present at image locations where the tracker does not detect any object. This way, the detector is only applied fully when it is necessary to complement or correct the tracker’s output, reducing computational cost while keeping segmentation performance high.
- The second identified inefficiency is that the tracker uses a relatively large memory, with object information from many previous frames, to re-identify new objects. The paper observes that the model only requires such a large memory in case of complex dynamics or motion between frames, which only happens rarely. Therefore, the paper uses a ‘lazy tracker’ by default, which uses a memory with fewer frames, and only uses the tracker with the large memory when the lazy tracker produces outputs that are divergent or inconsistent with previous-frame outputs. By using the less-expensive lazy tracker on most frames and only using the large-memory tracker when necessary, computational cost is reduced while still being able to use the large-memory tracker when necessary to prevent tracking errors.
-
The method’s effectiveness is adequately demonstrated through experiments. On the SA-Co/VEval benchmark, which is SAM3’s main evaluation benchmark for video-based concept segmentation, FastSAM3 achieves a 1.77x speedup compared to the original SAM3, while only obtaining slightly worse segmentation performance (e.g., 49.8 to 49.5 and 29.1 to 28.0 on the YT-Temporal-1B val and SA-V val subsets, respectively). The efficiency improvement at minimal accuracy drop makes FastSAM3 more practically useful for applications than SAM3.
Critical weaknesses
The paper does not have critical weaknesses that should cause it to be rejected. However, there are some smaller points for improvement, which are listed in the ‘Questions for Authors’ section. Considering both the strengths and these smaller points for improvement, I initially give a ‘Weak Accept’ rating. If the authors adequately address the points for improvement, I am willing to upgrade my rating to ‘Clear Accept’.
Questions for authors
There are several smaller points for improvement that could be addressed to further strengthen the paper. If the authors adequately address these points, I’m willing to upgrade my rating.
-
While the paper is generally clearly written, it contains a substantial number of typos and grammatical errors. The paper’s readability would be improved if these were fixed. Some (non-exhaustive) examples:
- L025: “thereby selectively trigger” => “thereby selectively triggering”
- L053: “it provide” => “it provides”
- L095: “that leverage” => “that leverages”
- L203: “2-th” => “2nd”
- L206: “where memory feature fusing historical image features” is grammatically incorrect
- L225: “can partitions” => “can partition” or “partitions”
- L229: “Then,We” => “Then, we”
- L286: “overleap” => “overlap”
- L263-L264: “which adopt ... and switch” => “which adopts and switches”
- L325 & L342: “more metric” => “more metrics”
-
In Fig. 1, it is not clear how the ‘detection score’ and ‘tracking score’ are computed. Is this detection score the same as the ‘entrant score’ (Eq. 11), and is the tracking score the same as the divergence score (L304)? The paper does not explicitly explain this. As the low values for these scores across video frames are used to demonstrate the detector’s and tracker’s redundancy and to motivate the method in the introduction (L052-L056), it is crucial that the meaning/definition of these scores is clearly specified when they are referred to. The fact that this is currently not done limits the understandability of the paper’s motivation.
-
In the ‘Memory Bank in Tracker’ paragraph (L205-L215), the paper uses to denote the tracker’s memory (L207, Eq. 6). However, earlier in the subsection, was used to denote the output of the tracker (Eq. 2), and was used to denote the memory features (L187). This is inconsistent and confusing, and should be fixed.
-
As shown in Tab. 4, the benefit of the temporal divergence score and the identity divergence score is negligible when also using the prediction divergence score to determine whether the tracker should use the sparse or dense memory. In L279-L298, the paper presents the two former divergence scores as being crucial, but in reality using only the prediction divergence appears to be sufficient. The paper should motivate why the temporal divergence score and the identity divergence score are still relevant and necessary, and why it would not be better to just use the prediction divergence score only, potentially yielding an even greater speedup because the other two divergence scores no longer need to be calculated.
-
From Tab. 1 (L327) and L377-L378, it is not fully clear whether the overall 1.77x speedup truly reflects the speedup of the end-to-end model, from input to output, including the backbone, or whether some components are excluded like for the component-specific speedups of 2.17x and 1.57x. The paper should more clearly explain the measurement setup and motivate this choice.
Note after rebuttal: the authors have answered my questions and addressed my minor concerns, so I upgrade my rating from 'weak accept' to 'clear accept'
6 comments
Response to Reviewer KWph (2/2)
Q4: As shown in Tab. 4, the benefit of the temporal divergence score and the identity divergence score is negligible when also using the prediction divergence score to determine whether the tracker should use the sparse or dense memory. In L279-L298, the paper presents the two former divergence scores as being crucial, but in reality using only the prediction divergence appears to be sufficient. The paper should motivate why the temporal divergence score and the identity divergence score are still relevant and necessary, and why it would not be better to just use the prediction divergence score only, potentially yielding an even greater speedup because the other two divergence scores no longer need to be calculated.
R4: Roles of temporal and identity divergence. [Q4]
Thank you for your valuable question. While prediction divergence leverages model's distractor awareness to identify disagreement among candidate masks, temporal and identity divergence capture motion inconsistency and cross-object identity conflicts, respectively.
To further examine their contributions, we extend the ablation study to other dataset splits. As shown in the following table, incorporating all three signals generally improves performance over prediction-only gating, with more noticeable gains on SA-V. Although the improvements are modest and vary across datasets, these results suggest that the additional signals provide complementary benefits.
| Dataset | Gating Signals | cgF1 | pmF1 | pHOTA | pDetA |
|---|---|---|---|---|---|
| YT1B_val | Prediction only | 49.1 | 55.9 | 70.1 | 59.9 |
| All | 49.4 | 56.3 | 70.1 | 59.9 | |
| YT1B_test | Prediction only | 49.8 | 56.2 | 69.2 | 59.5 |
| All | 49.8 | 56.2 | 69.3 | 59.5 | |
| SA-V_val | Prediction only | 28.4 | 43.4 | 60.3 | 44.4 |
| All | 28.9 | 43.9 | 60.6 | 44.6 | |
| SA-V_test | Prediction only | 29.3 | 42.6 | 57.6 | 40.5 |
| All | 29.7 | 42.9 | 57.6 | 40.5 | |
| SmartGlasses_val | Prediction only | 32.1 | 42.2 | 59.0 | 44.9 |
| All | 32.3 | 42.3 | 59.3 | 45.4 | |
| SmartGlasses_test | Prediction only | 34.7 | 44.4 | 62.5 | 49.1 |
| All | 35.3 | 45.1 | 62.6 | 49.1 |
We also analyze the triggering statistics of individual signals across three test sets. Prediction divergence and low confidence are the main triggers. Nevertheless, exclusive triggers from temporal and ID divergence account for 8.5%–18.9% of all dense-memory triggers, covering cases missed by the other conditions. Removing these signals reduces dense computation, but would also eliminate these independent checks, increasing the risk of tracking failures.
| Dataset | Trigger Condition | Trigger rate (%) | Exclusive trigger rate (%) | Trigger share (%) |
|---|---|---|---|---|
| YT1B_test | Prediction divergence | 6.72 | 1.89 | 69.81 |
| ID divergence | 0.83 | 0.13 | 8.63 | |
| Temporal divergence | 1.97 | 1.00 | 20.47 | |
| Low confidence | 6.42 | 1.54 | 66.69 | |
| Any | 9.63 | - | - | |
| SA-V_test | Prediction divergence | 3.94 | 1.20 | 62.34 |
| ID divergence | 0.27 | 0.06 | 4.35 | |
| Temporal divergence | 0.82 | 0.48 | 12.93 | |
| Low confidence | 4.50 | 1.73 | 71.23 | |
| Any | 6.32 | - | - | |
| SmartGlasses_test | Prediction divergence | 2.61 | 0.68 | 53.01 |
| ID divergence | 0.28 | 0.08 | 5.59 | |
| Temporal divergence | 1.12 | 0.85 | 22.67 | |
| Low confidence | 3.26 | 1.29 | 66.13 | |
| Any | 4.93 | - | - |
Q5: From Tab. 1 (L327) and L377-L378, it is not fully clear whether the overall 1.77x speedup truly reflects the speedup of the end-to-end model, from input to output, including the backbone, or whether some components are excluded like for the component-specific speedups of 2.17x and 1.57x. The paper should more clearly explain the measurement setup and motivate this choice.
R5: Clarification of speedup measurements and end-to-end latency. [Q5]
Thank you for pointing out this ambiguity. The 1.77× speedup refers to the combined detector and tracker branches, excluding backbone computation. We use this branch-level measurement to isolate the computational savings of EPG and DAG, which specifically optimize these two branches.
Table 6 in Appendix A.2 of the original submission provides a detailed latency breakdown, reproduced below. Including the backbone, full-pipeline latency decreases from 317 ms to 226 ms, corresponding to a 1.40× speedup. We have revised the manuscript to clearly distinguish the branch-level speedup (1.77×) from the full-pipeline speedup (1.40×), and corrected the corresponding descriptions to avoid ambiguity.
| Time (ms) | Backbone | Detector | Tracker-Memory | Tracker-Decoder | Others | Total | Detector+Tracker |
|---|---|---|---|---|---|---|---|
| SAM3 | 98 | 100 | 77 | 23 | 19 | 317 | 200 |
| LD-EPG | 95 | 46 (2.17x) | 72 | 17 | 19 | 249 | 135 |
| LT_DAG | 95 | 101 | 49 (1.57x) | 27 | 19 | 291 | 177 |
| Fast-SAM3 | 94 | 42 | 49 | 22 | 19 | 226 (1.40x) | 113 (1.77x) |
I thank the authors for their detailed response to my review.
With the response, the authors have partly addressed my concerns and answered my questions:
- Q1: The authors have fixed the textual errors that I included in my review as examples. However, multiple other errors still exist in the paper. Therefore, I do not consider this concern fully addressed. I would recommend the authors to do a more thorough proofread and fix the remaining errors. Some further examples (again not an exhaustive list of all errors):
- L064: "where detector's false positives" => "where the detector's false positives"
- L096: "to adaptively determines" => "to adaptively determine"
- L109: "For detector" => "For the detector"
- L300: "become low" => "becomes low"
- Q2: The authors have explained how the detection score and tracking score used in Fig. 1 are computed, and have added this explanation to the appendix. However, the explanation states that the detection score is the sum of the confidence scores of all new objects that are detected (which can be larger than 1.0), while the maximum value for the detection score in Fig. 1 is 1.0. This discrepancy is confusing and needs to be explained. Therefore, I also consider this question only partially answered.
- Q3: The authors have adequately addressed my concern about L205-L215 by fixing the notation.
- Q4: In their response, the authors have provided additional experiments to show that, in some experimental settings, the temporal divergence score and identity divergence score do improve the performance slightly more than in Tab. 4, compared to using only the prediction divergence score. While the performance improvements are still minimal, I can see that there is some benefit to the temporal and identity divergence scores. Therefore, I consider this concern addressed. However, I would encourage the authors to include the provided experimental results also in the appendix of the paper, so other readers also have access to this information.
- Q5: The authors have adequately answered my question by pointing out that the 1.77x speedup reflects only the speedup of the detector and tracker, and that that the overall model speedup is 1.40x. The authors have attempted to also clarify this in the paper by using the term 'dual-branch speedup' to indicate the 1.77x speedup. However, I believe it may not be immediately clear to the reader what 'dual-branch speedup' means. Also, I believe it would be relevant for the reader to be informed about the 1.40x overall model speedup in the main paper already, and not only in the appendix. Therefore, I would encourage the authors to better clarify in the paper what 'dual-branch speedup' means, and also mention the 1.40x overall model speedup in the main paper.
I've also carefully read the other reviews and the authors' responses to them, and I believe the comments in these reviews have mostly been properly addressed. I just have one follow-up question:
- In response R5 to reviewer YZog, the authors state that DAG can identify occlusion-induced tracking divergence. Could the authors explain how DAG achieves this, e.g., how it identifies a case where the tracker fails to re-identify an object that was occluded for a relatively large number of frames?
Since my concerns and questions have been addressed only partly, I keep my 'weak accept' rating unchanged.
Response to Reviewer KWph (1/2)
We sincerely appreciate your positive assessment and constructive suggestions, and are encouraged by your recognition of our work. We have carefully addressed the raised questions and revised the manuscript accordingly. All changes are highlighted in blue.
R1: Corrections to typos and grammar. [Q1]
Thank you for your careful review. We have thoroughly proofread the manuscript and corrected the typographical and grammatical errors to improve its readability and overall presentation.
R2: Definitions of the scores in Figure 1. [Q2]
Thank you for your valuable question. The Detection Score and Tracking Score in Fig. 1 are distinct from the entrant score and divergence scores used in our method. More specifically, they quantify the actual contributions of detection and dense memory to the model outputs at each frame, respectively, under the same historical state. Their computation is detailed below.
1. Detection Contribution Score. We quantify the contribution of detection by comparing the outputs with and without executing the detection branch under the same historical state. Three cases are considered: (1) if no objects are detected, the contribution is 0; (2) if no new objects are detected, the contribution is measured by the maximum confidence-weighted mask discrepancy between the two outputs; and (3) if new objects are detected, the contribution is measured by the sum of their confidence scores. Formally,
\begin{cases} 0, & A_t=B_t=\varnothing,\\\\[2mm] \displaystyle\max_{i\in A_t} s_{t,i}\left[1-\operatorname{IoU} \left(M_{t,i}^{D},M_{t,i}^{\neg D}\right)\right], & A_t=B_t\neq\varnothing,\\\\[3mm] \displaystyle\sum_{i\in A_t\setminus B_t}s_{t,i}, & A_t\neq B_t \land A_t\setminus B_t\neq\varnothing, \end{cases}$$ where $A_t$ and $B_t$ denote the object sets obtained with and without executing the detection branch, respectively. $M_{t,i}^{D}$ and $M_{t,i}^{\neg D}$ represent the corresponding object masks, and $s_{t,i}$ denotes the confidence score of object $i$ in the full detection output. **2. Full-Memory Contribution Score.** We quantify the contribution of full memory by comparing the tracking outputs obtained using full and sparse memory under the same historical state. The score is defined as the maximum IoU discrepancy between the resulting tracking masks: $$C_{\mathrm{FM}}(t)= \max_i\left[ 1-\operatorname{IoU} (M_{t,i}^{F},M_{t,i}^{S}) \right].$$ where $M_{t,i}^{F}$ and $M_{t,i}^{S}$ denote the binary tracking masks of object $i$ obtained using full and simplified memory, respectively, and $i$ indexes the tracked objects. When both masks are empty, their IoU is defined as $1$. For clarity, we have added the definitions and detailed computation procedures of both contribution scores in the Appendix A.5. The temporally sparse high values indicate that detection and dense memory contribute little in most frames, motivating adaptive execution. >Q3: In the ‘Memory Bank in Tracker’ paragraph (L205-L215), the paper uses $\mathcal{M}_t$ to denote the tracker’s memory (L207, Eq. 6). However, earlier in the subsection, $\mathcal{M}_t$ was used to denote the output of the tracker (Eq. 2), and $\mathcal{H}_t$ was used to denote the memory features (L187). This is inconsistent and confusing, and should be fixed. > **R3: Consistent notation for tracker outputs and memory features. [Q3]** Thank you for pointing out this notation inconsistency. We have revised the mathematical notation in the corresponding subsection, reserving $\mathcal{M}_t$ for the tracker outputs and consistently using $\mathcal{H}_t$ for the memory features. We have also checked the related equations and descriptions to ensure consistency throughout the paper.