- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Agentic Frame Selection (AFS) for memory management in SAM2-based 3D medical image segmentation. A policy network uses image and memory features, compressed through Spatial Squeezer modules, to select memory update actions. The policy is trained with group-normalized rewards and a clipped policy optimization objective. The authors evaluate the method on eleven medical segmentation datasets under interactive and five-shot in-context learning settings. The reported results and memory-budget experiments suggest that visual memory selection can improve segmentation and make better use of a limited memory bank.
Critical strengths
-
The paper addresses a useful limitation of FIFO memory updates: frames that remain relevant to later slices can be discarded solely because of their age. Selecting memories using visual content is a plausible and useful direction for improving context propagation in volumetric segmentation.
-
The method is evaluated across eleven medical datasets and two complementary settings, with qualitative comparisons and experiments varying memory-bank size. The reported improvements make the approach promising and support my weak-accept recommendation, while the reporting and practical-efficiency questions below need clarification.
Critical weaknesses
-
Numerical consistency and presentation need improvement. Table 2 reports average DSC/IoU of 69.4/59.8 for AFS, whereas Section 4.1 reports 68.8/59.4. Averaging the eleven displayed AFS dataset results gives approximately 70.41/60.88, so the table entries or aggregation procedure need clarification. Table 2 also gives DSC/IoU of 61.9/73.1 for FSSAM on Hippo; IoU should not exceed Dice under the same evaluation and aggregation protocol. The stated ICL improvement is 1.8% in the abstract and 1.9% in the contribution summary. These may be reporting or typographical errors, but corrected numbers are needed to establish the precise gains. The language and notation also require editing; in particular, the computation of the briefly defined oracle action a_i^* should be explained more explicitly.
-
The practical efficiency and scope of memory retention are insufficiently characterized. The paper does not measure policy-network latency or report end-to-end runtime and peak GPU memory including the Spatial Squeezer and Action Decoder. Using fewer stored frames alone does not establish an overall resource advantage, which matters for transfer to settings such as real-time segmentation. In addition, under the described action space, previously evicted frames do not appear to be reconsidered. Useful historical context can therefore still be lost if it becomes relevant again later. The conclusion acknowledges a restricted action set, and the practical implications of this limitation deserve clearer discussion.
Questions for authors
-
Could the authors reconcile the numerical inconsistencies between Table 2, Section 4.1, the abstract, and the contribution summary, and specify how dataset-level averages and Dice/IoU are computed? Please provide corrected values, including the FSSAM Hippo entry, and clarify whether these are typographical errors or differences in aggregation.
-
What are the per-frame latency of the policy network and the end-to-end inference time, throughput, and peak GPU memory of AFS versus FIFO on the same hardware? Please include the Spatial Squeezer and Action Decoder overhead and state the memory-bank sizes. Do savings from fewer memory frames outweigh this overhead, particularly for real-time segmentation?
-
Section 3.2 briefly defines a_i^* as the most important action in the current candidate set, with importance measured by segmentation Dice. How is it computed in practice? Is every candidate evaluated by updating a copy of the same memory bank, segmenting the current frame, and comparing the prediction with its ground-truth mask? Please clarify whether all candidates or a sampled subset are evaluated, when this oracle computation is used, and how test-time actions are selected without ground-truth masks.
-
Can previously evicted frames be re-encoded or reintroduced into the memory bank? If recovery is not supported, please discuss what happens when earlier anatomical context becomes useful again after a long slice gap, and the resulting limits of the context-retention claim. A recovery/retrieval mechanism could be an extension for future work; my current question is to clarify the capabilities and limitations of the submitted method.
2 comments
As the discussion period is drawing to a close, we wanted to kindly check if our previous response and the updated manuscript have satisfactorily addressed your concerns. We would greatly appreciate any further feedback or questions you might have, and we are happy to provide additional clarifications if needed.
Thank you again for your time and thoughtful review!
We thank the reviewer for acknowledging that “The paper addresses a useful limitation of FIFO memory updates ” and “The method is evaluated across eleven medical datasets and two complementary settings”
We appreciate the reviewer's careful evaluation of our work. Our responses follow.
W1. “Numerical consistency and presentation need improvement. ”
W2. “The practical efficiency and scope of memory retention are insufficiently characterized. ”
We presented the inference overhead of AFS below. Our AFS only 10% slower than SAM2
Q3: “a^*_i details”
Yes, we evaluate the candidates by updating a copy of the same memory bank, segmenting the current frame, and comparing the prediction with its ground-truth mask. In our experiments, we evaluate all candidates for the oracle. The oracle is meant for experiment only and in our setting, it cannot be computed without ground-truth.
Q4: “Can previously evicted frames be re-encoded or reintroduced into the memory bank? ”
The current AFS is not supporting re-encoded or reintroduced into the memory bank. We appreciate your suggestion for future improvement