acceptodds
Under review as a conference paper at ICLR 2027

FlashTAM: Target-Conditioned Memory Tokenization for On-Device Video Object Segmentation

Abstract

On-device promptable Video Object Segmentation (VOS) aims to track a user-specified target on device-side, imposing even stricter requirements on real-time responsiveness and model compactness. Although on-device VOS has been studied for years, existing methods typically face a trade-off among model size, inference latency, and segmentation quality, and therefore may not fully preserve the segmentation quality of SAM 2 . SAM 2 provides stronger target propagation through its streaming memory, but this memory mechanism also introduces substantial computational overhead, limiting its direct deployment on mobile devices. This creates a central challenge: retaining target-related temporal information for accurate propagation while organizing it into a compact memory representation suitable for efficient computation on mobile hardware. To overcome this challenge, we present a Flash Target Conditioned Memory (FlashTAM) design. FlashTAM comprises two main components: (1) Mask-guided ROI Localization (MRL). MRL reuses the predicted mask to localize a target-centric ROI on the memory grid, avoiding extra parameters or attention matrices. (2) Window-wise Adaptive Pooling (WAP). WAP adaptively pools the target-centric ROI into a near- regular token budget for efficient mobile attention. In summary, MRL decides Where to Sample and WAP decides How to Compress, jointly preserving target-relevant temporal information while reducing memory redundancy. Experiments on several benchmarks (SA-V, DAVIS-2017, MOSE, and YouTube-VOS 2019) validate that FlashTAM achieves a favorable accuracy-efficiency trade-off, notably reaching J&F on SA-V test and FPS on an iPhone 16 Pro, exceeding the second-best on-device method (EdgeTAM) by at least 1.2 J&F and 2.3 FPS.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Video Object Segmentation, On-Device Inference, SAM2, Target-Conditioned Memory Tokenization, Sparse Memory Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cMWzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FlashTAM, a memory tokenization approach for running promptable video object segmentation on mobile devices. It uses the predicted segmentation mask to identify and expand a target-centered bounding region, then divides this region into an aspect-ratio-aware grid. Each window contributes the memory feature corresponding to its highest mask-logit location. Integrated into a lightweight SAM 2-based architecture with feature distillation, the approach reduces memory attention costs and demonstrates a competitive balance between segmentation quality and on-device speed.

Critical strengths

The proposed memory selection procedure is simple and computationally inexpensive. It uses mask predictions already available in the pipeline, avoiding a separate selection module. Selecting one representative location per window also encourages broader spatial coverage than unrestricted top-k selection. The component ablations provide evidence that these choices benefit the resulting system.

The evaluation includes multiple VOS benchmarks and measurements on actual mobile hardware. Reporting end-to-end performance provides useful evidence of the method’s practical value.

Critical weaknesses

The methodological contribution is incremental relative to prior approaches to regional processing and memory compression. RMNet (CVPR 2021) restricts memory representations to regions containing the target, while RAVOS (TIP 2024) exploits object ROIs to avoid redundant segmentation and memory operations. EfficientTAM and EdgeTAM already investigate spatially organized memory compression, and Efficient-SAM2 selects salient memory tokens for efficient retrieval. The distinguishing elements of FlashTAM are an expanded bounding box derived from the predicted mask and maximum-logit sampling within local windows. Although reasonable, these choices largely combine established techniques, and the paper offers limited evidence of a broader methodological insight beyond this particular implementation.

The experimental comparisons do not fully isolate the benefits of the proposed memory construction from the surrounding system design. Comparisons with random, top-left, and global top-k sampling help justify the selection heuristic, but leave open how it performs against established compression methods with the same encoder, training procedure, distillation supervision, and memory budget. This is particularly relevant because feature distillation contributes to the final accuracy. Furthermore, constructing memory from predicted foreground regions can exclude useful information when masks underestimate the object or drift toward distractors. The paper acknowledges this vulnerability, but does not adequately characterize its consequences under occlusion, localization errors, or subsequent target reappearance.

Questions for authors

Could the authors evaluate ROI-restricted adaptive average pooling and representative existing memory compression modules using matched training, distillation, and token budgets?

How sensitive is the method to incomplete masks, extended occlusion, and similar-looking distractors? Under a fixed total token budget, would reserving some tokens for global context or background information improve recovery from tracking errors?

Reviewer_vVj9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlashTAM, an efficient video object segmentation framework built on SAM-2. It introduces mask-guided ROI localization and adaptive memory token pooling to reduce computational costs, alongside feature distillation to improve encoder representations. Experiments demonstrate competitive performance and 22.3 FPS on an iPhone 16 Pro, outperforming EdgeTAM in speed and SA-V accuracy. However, its methodological novelty is limited, improvements are inconsistent across benchmarks, and reliance on predicted masks raises concerns about error accumulation and robustness under challenging conditions.

Critical strengths
  1. The paper addresses a practical challenge in mobile video object segmentation by reducing the memory attention overhead of SAM-2. Its target-conditioned memory representation is well motivated by the varying relevance of historical tokens to the tracked object.
  2. FlashTAM introduces an efficient memory tokenization strategy without additional trainable networks or attention computation.
  3. The paper provides comprehensive evaluations across four established benchmarks, along with ablation studies on localization, token selection, token budgets, and feature distillation, offering empirical support for the proposed memory construction strategy.
Critical weaknesses
  1. The methodological novelty of FlashTAM is limited, as similar memory selection and compression strategies have been explored in Efficient-SAM2 (ICLR 2026), EfficientTAM (ICCV 2025), and EdgeTAM (CVPR 2025). Its combination of mask-guided ROI localization and window-wise token selection lacks sufficient differentiation from these existing approaches.
  2. FlashTAM relies heavily on previously predicted segmentation masks to determine which memory tokens are retained. However, inaccurate masks may exclude important target features from the memory bank, causing subsequent predictions to rely on incomplete or incorrect historical information. Such errors could accumulate over time and potentially lead to irreversible tracking failures. Although the authors acknowledge this limitation and introduce a small expansion margin, the paper lacks quantitative evaluations of error propagation, long-term tracking stability, and robustness to imperfect mask predictions.
  3. WAP assumes that higher mask confidence indicates greater importance for temporal memory matching, although foreground confidence does not necessarily reflect feature discriminability. This may discard informative boundary, appearance, or contextual cues. The lack of theoretical or empirical justification raises concerns about the effectiveness of the proposed token selection strategy.
  4. Although WAP ensures spatial coverage, selecting only one maximum-logit token per window may discard discriminative features, particularly near boundaries or under occlusion. The lack of feature diversity preservation and information loss analysis raises concerns about its ability to maintain target identity.

[1] Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval, ICLR 2026. [2] Efficient Track Anything, ICCV 2025. [3] EdgeTAM: On-Device Track Anything Model, CVPR 2025. [4] Recurrent Dynamic Embedding for Video Object Segmentation, CVPR 2022.

Questions for authors

Please see the Weaknesses.

1 comment

Thank you for the detailed response and additional experiments. These results provide useful empirical evidence for the effectiveness of FlashTAM. However, several of my original concerns remain only partially addressed.

W1.The results demonstrate that MRL+WAP outperforms several alternative memory selection strategies. However, improved performance does not necessarily establish methodological novelty. The proposed components still primarily rely on existing ROI localization and window-wise selection operations.

W2.The disappearance and reappearance experiments are useful but do not directly assess robustness to inaccurate mask predictions. Natural disappearance differs from ROI localization errors, which may accumulate over time and cause irreversible tracking failures. Controlled experiments with varying levels of mask corruption would provide more direct evidence of tracking robustness.

W3 and W4 have adequately addressed my concerns. I thank the authors for providing the additional experimental results.

Overall, the additional experiments strengthen the empirical evaluation, and I acknowledge the practical efficiency advantages of FlashTAM. But there are still some concerns remain insufficiently validated and need clarification.

Reviewer_7FvtOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents FlashTAM, an efficient memory tokenization method for on-device video object segmentation. The authors aim to reduce the memory attention overhead of SAM 2 while preserving segmentation quality. The proposed method consists of two components: Mask-guided ROI Localization (MRL), which uses the predicted object mask to identify a target-related region, and Window-wise Adaptive Pooling (WAP), which selects representative memory tokens within that region under a limited token budget. The authors also employ feature distillation from SAM 2.1-B+ to improve the lightweight image encoder.

Experiments on several video segmentation benchmarks show a favorable accuracy-efficiency trade-off. In particular, FlashTAM achieves 72.9 J&F on SA-V test and 22.3 FPS on an iPhone 16 Pro, outperforming EdgeTAM in both metrics.

Critical strengths
  • The paper addresses a practical and relevant problem. Reducing the computational cost of streaming memory is important for deploying video segmentation models on mobile devices. I appreciate that the authors focus on memory attention rather than only replacing the image encoder with a smaller backbone.

  • The proposed approach is simple and reasonably motivated. Using the predicted mask to guide memory selection is a natural way to retain target-related information, while window-based sampling helps maintain spatial coverage with a limited number of tokens. The method does not introduce an additional trainable selection network, which is appealing from a deployment perspective.

  • The on-device evaluation is a strong aspect of the paper. FlashTAM achieves better accuracy and throughput than EdgeTAM on an iPhone 16 Pro, despite using more memory attention layers. The experiments across several VOS benchmarks and the ablations on ROI localization and token selection are also useful.

Critical weaknesses
  • The methodological novelty is somewhat limited. The main components are based on relatively straightforward operations, including mask-based bounding box localization, ROI expansion, and window-wise token selection. Although their combination is effective, similar ideas of spatial pooling, token selection, and compact memory representations have already been explored in efficient SAM-based models. I would like to see a clearer explanation of what makes the proposed memory construction fundamentally different from existing approaches, beyond replacing global selection with mask-conditioned local sampling.

  • The contribution of memory tokenization is not fully separated from the overall training configuration. In Table 6, MRL+WAP achieves 71.3 J&F without feature distillation, while adding distillation improves it to 72.9. This 1.6-point gain is larger than the reported 1.2-point improvement over EdgeTAM on SA-V test. Although the ablations support the effectiveness of the proposed memory components, the comparison with EdgeTAM involves potentially different training configurations. A more closely matched comparison would help establish how much of the final improvement comes specifically from the memory tokenization strategy.

  • The benchmark selection and hyperparameter tuning protocol need clarification. Section 4.1 states that the ablations are evaluated on SA-V test, while Section 4.2 describes selecting the token budget according to segmentation quality across different budgets. It is therefore unclear whether the test set was used to choose the final token budget or ROI expansion margin. If so, this could affect the reliability of the reported test performance. I would appreciate clarification that these design choices were determined using a separate validation set before the final test evaluation.

  • The robustness of mask-guided memory selection deserves further analysis. Since the memory region is determined by the model's own prediction, inaccurate masks may exclude useful information and affect subsequent frames. This could be particularly problematic during occlusion, rapid object motion, target disappearance, or reappearance. The paper acknowledges this limitation, but the experiments do not provide a detailed analysis of these failure cases. I also noticed that the supplementary material specifies a fallback to full memory when the predicted mask is empty during training, while the corresponding inference behavior is less clear.

  • Some aspects of the efficiency evaluation could be strengthened. The iPhone results are useful, but the paper does not provide a detailed breakdown of the computational costs associated with memory selection, memory attention, and the remaining model components. Also, Efficient-SAM2 achieves stronger segmentation accuracy on SA-V test, but its inference efficiency is not compared under the same hardware setting. A more complete accuracy-latency comparison would help establish where FlashTAM stands among existing efficient memory designs.

Questions for authors
  • Could the authors provide a comparison with EdgeTAM or other memory compression strategies under matched backbone, training, and distillation settings? This would help isolate the improvement introduced by MRL and WAP.

  • Were the token budget and ROI expansion margin selected using SA-V test, or were they determined on a separate validation set? Could the authors clarify the complete hyperparameter selection protocol?

  • How does FlashTAM handle empty or severely inaccurate predicted masks during inference? In particular, can the model recover when a target disappears and later reappears after its memory representation has been compressed?

  • Could the authors provide a more detailed latency breakdown on the iPhone 16 Pro, including the overhead of MRL, WAP, and memory gathering? It would also be useful to report peak memory consumption.

  • How does the method perform on videos with small objects, heavy occlusion, or large target displacement? Does the benefit of target-conditioned memory selection remain consistent across these conditions?

Comments on the paper

1 comment
Area_Chair_UV5GOpenReview ↗

Discussion between Authors and Reviewers

Dear Reviewers,

The authors have submitted their rebuttal. Please read it and discuss with the authors via posted messages if necessary.

Note that the author-reviewer discussion phase ends on November 18 at 11:59 pm AoE. It is appreciated if you could share your thoughts as soon as possible, so the authors have enough time to clarify any remaining concerns.

Thank you!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.