acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Remember: Reinforcement-Guided Memory for SAM2 in 3D Medical Segmentation

Abstract

Recent medical foundation models such as Medical SAM2 have demonstrated strong performance for 3D multi-modal segmentation by propagating information across 2D slices via a memory bank. SAM2 uses a naive first-in-first-out memory, and discards informative long-range context. This leads to a problems, coined context mis-propagation: SAM2 is susceptible to mistakenly propagating the most recent patterns instead of mining past relevant contexts. To this end, we introduce Agentic Frame Selection (AFS), a novel reinforcement learning-based policy that dynamically selects important frames based on their expected contribution to segmentation performance. In both Interactive Segmentation and In-Context Learning settings that match various real-world clinical context, AFS demonstrates state-of-the-arts performance, improving upon recent works by 4.6% in Interactive Segmentation setting and 1.8% in In-Context Learning setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segmentation, Medical, SAM2

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F9sjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Agentic Frame Selection (AFS) for memory management in SAM2-based 3D medical image segmentation. A policy network uses image and memory features, compressed through Spatial Squeezer modules, to select memory update actions. The policy is trained with group-normalized rewards and a clipped policy optimization objective. The authors evaluate the method on eleven medical segmentation datasets under interactive and five-shot in-context learning settings. The reported results and memory-budget experiments suggest that visual memory selection can improve segmentation and make better use of a limited memory bank.

Critical strengths
  1. The paper addresses a useful limitation of FIFO memory updates: frames that remain relevant to later slices can be discarded solely because of their age. Selecting memories using visual content is a plausible and useful direction for improving context propagation in volumetric segmentation.

  2. The method is evaluated across eleven medical datasets and two complementary settings, with qualitative comparisons and experiments varying memory-bank size. The reported improvements make the approach promising and support my weak-accept recommendation, while the reporting and practical-efficiency questions below need clarification.

Critical weaknesses
  1. Numerical consistency and presentation need improvement. Table 2 reports average DSC/IoU of 69.4/59.8 for AFS, whereas Section 4.1 reports 68.8/59.4. Averaging the eleven displayed AFS dataset results gives approximately 70.41/60.88, so the table entries or aggregation procedure need clarification. Table 2 also gives DSC/IoU of 61.9/73.1 for FSSAM on Hippo; IoU should not exceed Dice under the same evaluation and aggregation protocol. The stated ICL improvement is 1.8% in the abstract and 1.9% in the contribution summary. These may be reporting or typographical errors, but corrected numbers are needed to establish the precise gains. The language and notation also require editing; in particular, the computation of the briefly defined oracle action a_i^* should be explained more explicitly.

  2. The practical efficiency and scope of memory retention are insufficiently characterized. The paper does not measure policy-network latency or report end-to-end runtime and peak GPU memory including the Spatial Squeezer and Action Decoder. Using fewer stored frames alone does not establish an overall resource advantage, which matters for transfer to settings such as real-time segmentation. In addition, under the described action space, previously evicted frames do not appear to be reconsidered. Useful historical context can therefore still be lost if it becomes relevant again later. The conclusion acknowledges a restricted action set, and the practical implications of this limitation deserve clearer discussion.

Questions for authors
  1. Could the authors reconcile the numerical inconsistencies between Table 2, Section 4.1, the abstract, and the contribution summary, and specify how dataset-level averages and Dice/IoU are computed? Please provide corrected values, including the FSSAM Hippo entry, and clarify whether these are typographical errors or differences in aggregation.

  2. What are the per-frame latency of the policy network and the end-to-end inference time, throughput, and peak GPU memory of AFS versus FIFO on the same hardware? Please include the Spatial Squeezer and Action Decoder overhead and state the memory-bank sizes. Do savings from fewer memory frames outweigh this overhead, particularly for real-time segmentation?

  3. Section 3.2 briefly defines a_i^* as the most important action in the current candidate set, with importance measured by segmentation Dice. How is it computed in practice? Is every candidate evaluated by updating a copy of the same memory bank, segmenting the current frame, and comparing the prediction with its ground-truth mask? Please clarify whether all candidates or a sampled subset are evaluated, when this oracle computation is used, and how test-time actions are selected without ground-truth masks.

  4. Can previously evicted frames be re-encoded or reintroduced into the memory bank? If recovery is not supported, please discuss what happens when earlier anatomical context becomes useful again after a long slice gap, and the resulting limits of the context-retention claim. A recovery/retrieval mechanism could be an extension for future work; my current question is to clarify the capabilities and limitations of the submitted method.

2 comments

We thank the reviewer for acknowledging that “The paper addresses a useful limitation of FIFO memory updates ” and “The method is evaluated across eleven medical datasets and two complementary settings”

We appreciate the reviewer's careful evaluation of our work. Our responses follow.

W1. “Numerical consistency and presentation need improvement. ”

  • The average numbers were misleading because we included an incomplete experiment. After fixing the number, ours still significantly outperforms SAM2 by 2.6% in few-shot settings across 11 datasets .
  • FSSAM's Hippo IoU (73.1) exceeds its Dice (61.9) because we mistakenly copied the number from our sheet to the paper in reversed order. We resolved and the updated numbers are FSSAM's Hippo Dice (73.1) and IoU (61.9).
  • The SAM2Rad invalid IoU on BTCV stem from inconsistent implementation in the author's released BTCV eval code. We double checked on all dataset and resolved the inconsistency, yielding the following results: SAM2Rad BTCV Dice 50.4, IoU 40.9.
  • Five-shot DICE is 69.4 in table 2 but 68.8 in text due to us inconsistently updating new experimental results to our manuscript. We verified and updated consistent numbers to our manuscript

W2. “The practical efficiency and scope of memory retention are insufficiently characterized. ”
We presented the inference overhead of AFS below. Our AFS only 10% slower than SAM2

ModelThroughput (frames/s)Inference time per volume (s/volume)Peak Memory (GB)
SAM27.43.51.1
AFS6.65.23.0

Q3: “a^*_i details”
Yes, we evaluate the candidates by updating a copy of the same memory bank, segmenting the current frame, and comparing the prediction with its ground-truth mask. In our experiments, we evaluate all candidates for the oracle. The oracle is meant for experiment only and in our setting, it cannot be computed without ground-truth.

Q4: “Can previously evicted frames be re-encoded or reintroduced into the memory bank? ”
The current AFS is not supporting re-encoded or reintroduced into the memory bank. We appreciate your suggestion for future improvement

As the discussion period is drawing to a close, we wanted to kindly check if our previous response and the updated manuscript have satisfactorily addressed your concerns. We would greatly appreciate any further feedback or questions you might have, and we are happy to provide additional clarifications if needed.

Thank you again for your time and thoughtful review!

Reviewer_43HvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Agentic frame seleciton, a RL-based method for managing SAM2's memory in 3D medical image segmentation. Instead of retraining the most recent slices, it uses visual features to learn which memory frame needs to be kept or to replace.

The method compresses spatial features for efficient action selection and is evaluated under inital box prompting and five shot in-context learning on multiple medical image datasets.

Critical strengths
  1. this paper has a clear motivation, it selects which slices need to be retained in the memory bank and which needs to be replaced.
  2. the memory policy considers visual features, allowing retention decisions to depend on image content instead of just the recent slices.
  3. the feature compression module addresses a practical challenge in using large spatial representations for memory selection.
  4. The memory budget experiments examine the central hypothesis, whether selecting useful slices improves segmentation performance with limited memory budget.
Critical weaknesses
  1. I feel this paper looks rushed, several reported results are inconsistent. such as five-shot DICE is 69.4 in table 2 but 68.8 in text, while the displayed dataset scores average looks like around 70. Thus i feel concerned about the reliability of the numbers shown in this paper.
  2. A lot of experimental details are missing, such as how the data split, how the reward configuration, and seems no appendix?
  3. The comparison do not clearly isolate their memory mechanism, Figure 4 says SAM2 is trainable, so improvement of results may purely comes from model adaptation?
  4. why SAM2RL is not included in the comparison results?
Questions for authors
  1. could the authors recheck the numbers for the entire paper?
  2. how is the complete reward designed, how is the additional penalty term?
  3. which SAM2 components are trained?
3 comments

We thank the reviewer for noting that “this paper has a clear motivation” and “A sensible, broadly evaluated design.”

We appreciate the reviewer's careful evaluation of our work. Our responses follow.

W1. Number Inconsistency

  • The average numbers were misleading because we included an incomplete experiment. After fixing the number, ours still significantly outperforms SAM2 by 2.6% in few-shot settings across 11 datasets.
  • Five-shot DICE is 69.4 in table 2 but 68.8 in text due to us inconsistently updating new experimental results to our manuscript. We verified and updated consistent numbers to our manuscript

W2. Experiment details

  • How is η in the reward defined?: η in Figure 3 out-of-date. Our final reward design only consists of the delta between 2 Dice. We updated the Figure 3 to be up-to-date
  • What train/test splits were used for the MSD tasks?: Since MSD test sets are not published, we split MSD training set into train/test with 8:2 ratio.

W3. SAM2 learnable modules & AFS isolation experiment
In every experiment, including FIFO-based SAM2, we set SAM2’s mask decoder, memory encoder and memory attention to be learnable, while the remaining are frozen.

We did present the controlled, same-backbone, same-training comparison between FIFO-based SAM2 and AFS in Figure 5. Our proposed memory selection significantly attribute to the gain by 1.1%. Following your suggestion, we also updated the numbers in Table 1 for more clarity. The results are:

AverageSarcomaHeartLiverHippoSpleen
DiceDiceDiceDiceDiceDice
SAM262.364.456.359.577.271.7
Ours72.674.579.669.086.388.4
BTCVProstateLungPancreasHepaticColon
DiceDiceDiceDiceDiceDice
SAM274.266.958.752.455.247.0
Ours81.173.265.458.861.654.6

W4: “Why SAM2RL is not included in the comparison results?”
We did not include SAM2RL results because they are on arXiv, not published in peer-reviewed venues. Follow your suggestion, we additionally present SAM2RL results on few-shot setting below, ours still significantly outperform SAM2RL by 4.2%

AverageSarcomaHeartLiverHippoSpleen
DiceIoUDiceIoUDiceIoUDiceIoUDiceIoUDiceIoU
SAM2RL66.657.876.566.685.778.166.559.478.768.491.687.0
Ours70.861.278.668.186.378.868.260.978.868.192.087.8
BTCVProstateLungPancreasHepaticColon
DiceIoUDiceIoUDiceIoUDiceIoUDiceIoUDiceIoU
SAM2RL71.363.157.546.949.540.356.746.454.442.744.637.2
Ours74.466.364.653.659.246.257.646.855.543.763.752.5
Reviewer_43HvOpenReview ↗

Seems this paper is not ready yet.

I thank the authors for their rebuttal and for addressing some of my concerns. However, I would appreciate clarification on two remaining points.

First, could the authors clarify whether the missing numerical results in the initial submission were due to experiments that had not yet been completed? I also get confused that the value of η in Figure 3 was outdated, so why was that in the submission? These issues raise concerns about whether the experimental results and presentation were sufficiently complete and checked at the time of submission.

Second, I recently read the comments from the original authors of SAM2RL. It would be helpful to hear the authors’ response to the concerns raised, including whether they identify valid issues and, if so, how these affect the reported results and conclusions.

Q1a. First, could the authors clarify whether the missing numerical results in the initial submission were due to experiments that had not yet been completed?

We did not have any missing numerical results. The inconsistent Average number was due to the Excel formula including the outdated experiment results. These outdated results are not in the paper.

Q1b. I also get confused that the value of η in Figure 3 was outdated, so why was that in the submission?

We once included η as a hyper-parameter of our method. Yet after ablation, we find that it has no positive effect. Thus, we completely remove it from the text and the final method. We acknowledge that we forget to clean up η from Figure 3. We have update Figure 3 to correctly reflect our final method.

Q2. Second, I recently read the comments from the original authors of SAM2RL. It would be helpful to hear the authors’ response to the concerns raised, including whether they identify valid issues and, if so, how these affect the reported results and conclusions.

We responded to SAM2RL authors. The SAM2RL authors raise a valid point that SAM2RL do not aim to solve generalizability.

SAM2RL overfits a different policy for each volume for theoretical analysis, and cannot be used to infer and evaluate on unseen and no-ground-truth volumes in test set. Meanwhile, ours train a single policy across volumes to achieve empirically generalizable and efficient SAM2 memory selection, working on unseen test volumes. As such, we adapted SAM2RL by using a single RL agent to evaluate on unseen test sets.

The points raised do not affect the reported results and conclusions: our RL-driven AFS achieves better generalizability than SAM2RL.

We update the manuscript to describe the purpose of SAM2RL in theoretical analysis and contrast against our empirical generalizability purpose.

Reviewer_cNoxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Agentic Frame Selection (AFS), a reinforcement learning-based memory management method for SAM2 in 3D medical image segmentation. It replaces SAM2's first-in-first-out memory eviction with a learned policy that decides which slices to keep, motivated by what the authors call informative context dropping and context mis-propagation, which they quantify with a new Context Drop metric. The policy uses a Perceiver-based Spatial Squeezer to compress image and memory features and an Action Decoder to select eviction actions, and it is trained with a clipped surrogate objective, group-normalized advantages, and a per-slice Dice-improvement reward. Evaluated on 11 datasets in an interactive setting and a 5-shot in-context learning setting, AFS reports higher average Dice than SAM2-based variants and fully supervised baselines such as nnU-Net and MedNeXt, and reaches stronger performance with smaller memory banks than FIFO-based SAM2.

Critical strengths
  • A well-motivated problem made measurable. For volumetric data, slice order does not guarantee relevance, making FIFO eviction in SAM2-style memory a natural target for improvement. The Context Drop metric turns this into something quantifiable by comparing each eviction decision against the best available choice at that step. This is consequential because it is the paper's most transferable contribution, offering the community a simple oracle-gap diagnostic for any memory policy, independent of whether AFS itself holds up.

  • A sensible, broadly evaluated design. Unlike index-only memory controllers, the policy processes visual content efficiently using Perceiver-based compression of memory features. The method is evaluated in two clinically relevant settings (interactive and 5-shot in-context) across 11 datasets, with Table 1 showing consistent gains on every dataset. This matters to my assessment because such breadth suggests the idea is not overfitted to a single benchmark. If the soundness concerns were resolved, this evidence would heavily support acceptance.

Critical weaknesses
  • The gains are not attributed to memory selection. Table 1 lacks a controlled, same-backbone, same-training comparison between standard FIFO-based SAM2 and AFS. The sole direct comparison appears in the memory-size ablation (Figure 5), where the baseline numbers are difficult to reconcile with the rest of the manuscript: FIFO-based SAM2 achieves 69.0 Dice on Heart in the interactive setting (using full training and box prompts), yet reaches 85.5 on Heart under 5-shot ICL (Table 2). Furthermore, Figure 4 suggests that SAM2 components are fine-tuned alongside the policy, obscuring whether the baselines received equivalent training. The evaluation also omits straightforward non-FIFO heuristics (such as similarity- or confidence-based selection used in Medical SAM 2), SAM2RL results, and the greedy oracle. These comparisons are necessary to justify the need for a learned policy. This weakness is decisive because the paper’s central claim rests on learned eviction driving performance, and the current experiments fail to isolate this effect from confounding training differences.

  • The reported results are internally inconsistent and lack variance. The Table 2 average column does not match the mean of the per-dataset values for several rows. SAM2Rad, for example, is reported at 57.7 against a computed 52.1. Two baseline cells are mathematically impossible: FSSAM's Hippo IoU (73.1) exceeds its Dice (61.9), and SAM2Rad's BTCV IoU (21.5) is below the minimum consistent with its Dice. The headline ICL gain is stated as +1.8 in the abstract and +1.9 in the contributions, and neither figure matches the tables. The +4.6 interactive gain is measured against SAM2-Adapter rather than the strongest baseline, over which the margin is +0.6. No seeds or variance are reported. This is consequential because the margins over the best baselines are only about 0.6–2 Dice, small enough that transcription errors or run-to-run variation could account for them. As a result I can't treat the state-of-the-art claims as established.

Questions for authors

Clarifications

  1. Which SAM2 components are trained together with the AFS policy? Fig. 4 appears to mark memory attention, the decoder and the memory encoder as trainable. Was the FIFO SAM2 baseline in Fig. 5 fine-tuned with the same data, schedule and components? Please also explain why FIFO SAM2 reaches 69.0 Dice on Heart in the interactive setting (full training plus a box prompt), but 85.5 in the 5-shot ICL setting (Table 2).

  2. Can you report one comparison on the same fine-tuned backbone that includes FIFO, simple non-FIFO heuristics (for example similarity- or confidence-based selection, as in Medical SAM 2's self-sorting memory), SAM2RL, the greedy oracle a* and AFS? Please give mean ± std over several seeds. This would directly test whether the reported gains come from learned eviction.

  3. The Table 2 average column doesn't match the per-dataset means for several rows (SAM2Rad, FSSAM, SANSA, Ours). Two cells are impossible: FSSAM Hippo has IoU above Dice, and SAM2Rad BTCV has IoU below the minimum its Dice allows. Which values are correct? Were all baselines rerun under your protocol or taken from prior work? How were SAM2-Long and DAM-4SAM adapted or fine-tuned? Which nnU-Net and MedNeXt implementations were used, given that the citations point to UNet++ and SAM2-Adapter?

  4. What are its units? Fig. 2 values reach about −2.8, which is impossible for Dice differences on a [0, 1] scale. Was it computed on held-out test volumes? How was SAM2RL trained for this comparison, given that it was originally evaluated with a separate agent per video?

  5. Method and protocol details.

    • How is η in the reward defined?
    • Which action space is correct: the one in Sec. 3.2 (always discard one frame) or the one in Fig. 3 (which includes no-op and add-without-drop)?
    • Is the objective purely one-step, with no returns?
    • What train/test splits were used for the MSD tasks?
    • How are multi-class datasets such as BTCV, Prostate and Hippocampus handled?
    • What is the inference overhead of the policy per slice?

Suggestions

  • Revise the framing that SAM2-based medical models use naive FIFO, and discuss existing non-FIFO memory designs such as Medical SAM 2 and CRISP-SAM2.
  • Consider multi-step returns, since the long-range-context motivation suggests eviction decisions have delayed effects.
  • Fix the mis-cited references and the GRPO description.
  • Add the missing appendix with hyperparameters and training cost, and release code.
2 comments

We thank the reviewer for noting that “this paper has a clear motivation” and “A sensible, broadly evaluated design.”

We appreciate the reviewer's careful evaluation of our work. Our responses follow.

W1. “several reported results are inconsistent”

  • The average numbers were misleading because we included an incomplete experiment. After fixing the number, ours still significantly outperforms SAM2 by 2.6% in few-shot settings across 11 datasets.
  • Five-shot DICE is 69.4 in table 2 but 68.8 in text due to us inconsistently updating new experimental results to our manuscript. We verified and updated consistent numbers to our manuscript

W2. “A lot of experimental details are missing ”

  • reward configuration : η in Figure 3 out-of-date. Our final reward design only consists of the delta between 2 Dice. We updated the Figure 3 to be up-to-date
  • train/test splits were used: We split all dataset into train/test split in 8:2 ratio. Since MSD test sets are not publicly available, we split the training set into our train 8:2 train/test split.
  • Follow your suggestion, we updated our appendix with computation overhead of our AFS
ModelThroughput (frames/s)Inference time per volume (s/volume)Peak Memory (GB)
SAM27.43.51.1
AFS6.65.23.0

W3. “The comparison do not clearly isolate their memory mechanism”
In every experiment, including FIFO-based SAM2, we set SAM2’s mask decoder, memory encoder and memory attention to be learnable, while the remaining are frozen.

We did present the controlled, same-backbone, same-training comparison between FIFO-based SAM2 and AFS in Figure 5. Our proposed memory selection significantly attribute to the gain by 1.1%. Following your suggestion, we also updated the numbers in Table 1 for more clarity. The results are:

AverageSarcomaHeartLiverHippoSpleen
DiceDiceDiceDiceDiceDice
SAM262.364.456.359.577.271.7
Ours72.674.579.669.086.388.4
BTCVProstateLungPancreasHepaticColon
DiceDiceDiceDiceDiceDice
SAM274.266.958.752.455.247.0
Ours81.173.265.458.861.654.6

W4: “Why SAM2RL is not included in the comparison results?”
We did not include SAM2RL results because they are on arXiv, not published in peer-reviewed venues. Follow your suggestion, we additionally present SAM2RL results on few-shot setting below, ours still significantly outperform SAM2RL by 4.2%

AverageSarcomaHeartLiverHippoSpleen
DiceIoUDiceIoUDiceIoUDiceIoUDiceIoUDiceIoU
SAM2RL66.657.876.566.685.778.166.559.478.768.491.687.0
Ours70.861.278.668.186.378.868.260.978.868.192.087.8
BTCVProstateLungPancreasHepaticColon
DiceIoUDiceIoUDiceIoUDiceIoUDiceIoUDiceIoU
SAM2RL71.363.157.546.949.540.356.746.454.442.744.637.2
Ours74.466.364.653.659.246.257.646.855.543.763.752.5

As the discussion period is drawing to a close, we wanted to kindly check if our previous response and the updated manuscript have satisfactorily addressed your concerns. We would greatly appreciate any further feedback or questions you might have, and we are happy to provide additional clarifications if needed.

Thank you again for your time and thoughtful review!

Comments on the paper

3 comments
~Tomáš_Čížek1OpenReview ↗

Questions on the use and description of SAM2RL

Dear authors,

I am one of the authors of SAM2RL (Adamyan et al., 2025), which you use as a baseline. I have read your paper carefully and have a few questions and remarks.

First, SAM2RL was never meant to generalize to unseen videos. Instead, it estimates an upper bound on the performance of memory control in SAM 2 by overfitting separate policies to individual videos from the SA-V dataset. Its simplistic input features are therefore intentional, as they are sufficient for fitting individual video sequences.

Given this, my questions are:

  • What is the purpose of comparing with SAM2RL in Figure 2? SAM2RL yields a set of policies intentionally overfitted to individual SA-V videos. They cannot be expected to work on unseen videos, and were never meant to.
  • How exactly do you evaluate SAM2RL in Figure 2? We trained 64 policies, each overfitted to exactly one of 64 SA-V videos. How do you pair these policies with the volumes/videos in your benchmarks? The correct use of SAM2RL would be to overfit a new policy to each of them to obtain the upper bound. Is this what you did? In that case, with the same SAM 2 weights and memory size as your method, SAM2RL should match or even exceed it.
  • Could you please release the code of your SAM2RL reimplementation? We have never released our code or trained policies, so I assume you reimplemented the method. Releasing your implementation would clarify the missing training and evaluation details.
  • Could you please describe the purpose of SAM2RL correctly? The paper never explains what SAM2RL was designed for and compares it as if it were a fully usable tracker. It lists two limitations of SAM2RL: “First, SAM2RL overlooks the context mispropagation phenomenon, leaving the mechanistic reasons behind its RL gains unexplained. Second, its policy network is blind to visual features.” This misinterprets the intention of the method.
  • “Concurrent work”. You call SAM2RL a concurrent work, but it has been on arXiv since 11 July 2025, more than a year before the submission. Could you please correct this?

We would like to express our sincere gratitude to all reviewers for their constructive feedback and are excited that the reviewers endorse our works for: identifying a well-motivated problem (cNox, 43Hv, F9sj) and extensive evaluation on broad-range of datasets and settings (cNox, F9sj)

Here, we provide responses to common questions raised by the reviewers. In addition, we respond to each individual reviewer for their unique questions.

Clarification: Number Inconsistency (cNox, 43Hv, F9sj)
We verified and updated all relevant numbers in Table 2, Abstract and Results section to resolve the inconsistencies as follow

  • The average numbers were misleading because we included an incomplete experiment. After fixing the numbers, ours still significantly outperforms SAM2 by 2.6% in few-shot settings across 11 datasets.
  • FSSAM's Hippo IoU (73.1) exceeds its Dice (61.9) because we mistakenly copied the number from our sheet to the paper in reversed order. We resolved and the updated numbers are FSSAM's Hippo Dice (73.1) and IoU (61.9).
  • The SAM2Rad invalid IoU on BTCV stem from inconsistent implementation in the author's released BTCV eval code. We double checked on all dataset and resolved the inconsistency, yielding the following results: SAM2Rad BTCV Dice 50.4, IoU 40.9.
  • Our +4.6 gain is more specifically over other SAM2 methods, which emphasize how much our method improves SAM2.
  • Five-shot DICE is 69.4 in table 2 but 68.8 in text due to us inconsistently updating new experimental results to our manuscript. We verified and updated consistent numbers to our manuscript

Clarification: SAM2 learnable modules & AFS isolation experiment (cNox, 43Hv)
To clarify our experiment setting, we updated the experiment's description in Sec 4.1 and add SAM2 results into Table 1.

In every experiment, including FIFO-based SAM2, we set SAM2’s mask decoder, memory encoder and memory attention to be learnable, while the remaining are frozen.

We did present the controlled, same-backbone, same-training comparison between FIFO-based SAM2 and AFS in Figure 5. Our proposed memory selection significantly attribute to the gain by 1.1%. Following your suggestion, we also updated the numbers in Table 1 for more clarity. The results are:

AverageSarcomaHeartLiverHippoSpleen
DiceDiceDiceDiceDiceDice
SAM262.364.456.359.577.271.7
Ours72.674.579.669.086.388.4
BTCVProstateLungPancreasHepaticColon
DiceDiceDiceDiceDiceDice
SAM274.266.958.752.455.247.0
Ours81.173.265.458.861.654.6

Clarification: Experiment details (cNox, 43Hv)
To clarify our experiments, we add these details into the our method description in Sec 3.3 and experiments’ details in Sec 4

  • “How is η in the reward defined?”: η in Figure 3 out-of-date. Our final reward design only consists of the delta between 2 Dice. We updated the Figure 3 to be up-to-date
  • “Which action space is correct the one in Sec. 3.2 (always discard one frame) or the one in Fig. 3 (which includes no-op and add-without-drop)?”: In our design, no-op is equivalent to discarding the incoming frame (i.e. frame i-1 awaiting decision). Meanwhile, add-without-drop is common for both FIFO SAM2 and AFS, and it only happens for the first few frames, we omit it from Sec. 3.2 experiment.
  • “Is the objective purely one-step, with no returns?”: Yes, SAM2 roll-out is prohibitively expensive, thus we employ one-step reward and find the results promising.
  • “What train/test splits were used for the MSD tasks?”: Since MSD test sets are not published, we split MSD training set into train/test with 8:2 ratio.
  • “How are multi-class datasets such as BTCV, Prostate and Hippocampus handled?”: during inference, the target segmentation class is represented by prompts in interactive setting and examples in few-shot setting. We run inference on each class separately, then compute per-class metrics before averaging them for per-dataset metrics.

Experiment: FIFO-based SAM2 vs AFS efficiency (cNox, F9sj)
To clarify the computation overhead of AFS, we add the following results into our Appendix.

Our AFS only 10% slower than SAM2

ModelThroughput (frames/s)Inference time per volume (s/volume)Peak Memory (GB)
SAM27.43.51.1
AFS6.65.23.0

Q1: What is the purpose of comparing with SAM2RL in Figure 2? SAM2RL yields a set of policies intentionally overfitted to individual SA-V videos. They cannot be expected to work on unseen videos, and were never meant to.

Response: Since we design a novel RL-driven memory selection in SAM2, we compare with SAM2RL for completeness. We agree that the original SAM2RL agents are overfitted to single videos and are not meant for unseen data, as you pointed out. SAM2RL is orthogonal to ours. Sam2rl aims to find the theoretical upper-bound by overfit multiple policy networks. In contrast, we focus on finding an empirically generalizable and efficient method: creating a single RL-driven policy that can generalize across videos/volumes for SAM2 memory selection. Our benchmark aims to empirically evaluate the generalizability of the models: whether training a single RL agent to learn an optimal action enable generalizability of SAM-2 memory selection.

We agree that the original SAM2RL agents are overfitted to single videos and are not meant for unseen data. The SAM2RL results in Fig. 2 are not from the original implementation. We modified the Fig. 2 caption to note SAM2RL-inspired model and refer the to the SAM2RL implementation details in Sec 4.

Q2: How exactly do you evaluate SAM2RL in Figure 2? We trained 64 policies, each overfitted to exactly one of 64 SA-V videos. How do you pair these policies with the volumes/videos in your benchmarks? The correct use of SAM2RL would be to overfit a new policy to each of them to obtain the upper bound. Is this what you did? In that case, with the same SAM 2 weights and memory size as your method, SAM2RL should match or even exceed it.

Response: To enable generalization to unseen volumes with varied length, we re-implement SAM2RL as a single Policy Network instead of 64 policies. We fix the binary vector dimension to the maximum length. We train SAM2RL together with SAM2 backbone on train set, then test on unseen volumes in test set. We include these details in our Sec. 4 to clarify our re-implementation to answer the our key generalizability research question as discussed in Q1.

Q3: Could you please release the code of your SAM2RL reimplementation? We have never released our code or trained policies, so I assume you reimplemented the method. Releasing your implementation would clarify the missing training and evaluation details.

Response: We will release the SAM2RL codebase upon acceptance to reduce the chance of leaking identity. The summary of our reimplementation is as mentioned above.

Q4: Could you please describe the purpose of SAM2RL correctly? The paper never explains what SAM2RL was designed for and compares it as if it were a fully usable tracker. It lists two limitations of SAM2RL: “First, SAM2RL overlooks the context mispropagation phenomenon, leaving the mechanistic reasons behind its RL gains unexplained. Second, its policy network is blind to visual features.” This misinterprets the intention of the method.

Response: We thank the authors for raising this point. As such, we rewrite the paragraph to correctly describe the purpose of SAM2 as follows: "By overfitting one policy per video, SAM2RL shows that learned memory control can beat heuristic rules. However, SAM2RL does not take visual features as input in contrast to our AFS, thus SAM2RL does not solve context mis-propagation."

Q5: “Concurrent work”. You call SAM2RL a concurrent work, but it has been on arXiv since 11 July 2025, more than a year before the submission. Could you please correct this?

Response: Yes, we updateed SAM2RL to “prior work” in our manuscript, following your suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.