- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes MedCore, a structured pruning method to reduce MedSAM’s model size while preserving segmentation boundaries. It selects attention heads and MLP groups to prune by estimating the effect of removing them or resetting their weights to the original SAM checkpoint. Thet also propose boundary-weighted Fisher scores to guide prune, followed by recovery training and continual fine-tuning. Experiments show that they can achieve a much compressed model with almost no change on DICE.
Critical strengths
- The motivation is clear: they use pruning methods to preserve model performance while compressing model sizes.
- Their proposed boundary leverage part is interesting; I think I didn't see similar work before.
- Their matched pruning comparisons in the appendix and the component ablations provide useful evidence, particularly for the boundary-aware Fisher scoring.
Critical weaknesses
- All experiments use one SAM-MedSAM ViT-B checkpoint pair. It is unclear whether these findings are medical image tasks generalizable or just this particular model pairing.
- The strongest pruning methods are in appdenix and don't include final fine-tuning. It is less convincing to the advantage over existing methods bc the main tables in paper are just MedSAM vs. compressed model.
- I think the boundary scoring idea is interesting, but I feel the datasets the authors choose are less challenging in this task. Such as the visualization of the results in Figure 2 (btw, the authors don't explain which dataset it is in the figure title), it is a pretty regular round object; thus, it is hard to show the benefit of boundary accuracy. I think the results can be more convincing if the authors pick a more challenging dataset, whose boundary accuracy do matter.
- My major concern also includes the paper presentation. I feel the entire paper looks rushed, like inconsistent digits (4 digits for some metrics and even 6 digits for some others in Table 4); also inconsistent numbers reported (for example, 385.2 G vs. 377.9 G for h50_m70). Also, the h50_m70 is also like an intermediate config version of the model, which is hard to follow and understand during reading. I suggest the authors could spend more time polishing their writing and presentations.
Questions for authors
- Does the reset score remain useful for another backbone pair or backbone config?
- Do the final fains over the pruning baselines in Table 7 remain after the same fine-tuning?
- How does MedCore compare with a compact student distilled from MedSam under comparable training and inference budgets?
4 comments
Dear Reviewer,
Thank you again for your constructive feedback. As the discussion period closes on November 18, we remain available to provide any further clarification or address any remaining questions.
If you feel that our rebuttal has adequately addressed your concerns, we would greatly appreciate your reconsideration of the overall assessment.
Sincerely,
The Authors
Authors try to answer my questions but not solve any.
Thank you for the response and for clarifying the scope of this work.
However, my main concerns remain unresolved. the response acknowledges the issues i raised, but provide no concrete corrections or additional results (i just checked the latest paper pdf). For example, the inconsistent FLOPs values. the qualitativ e concern could also be addressed by finding the dataset and providing a more challenging example that illustrates boundary preservation.
These seem feasible to address directly, yet the rebuttal did any. I really wish the authors can spend more time on their paper, also think about ur response; (this also reflect my impression when reading this paper, feel very rushed and should spend more time to polish before submission).
I understand the rational for comparing pruning scores without fine-tuning. However, this does not answer whether MedCore's advantage remains when all pruning baselines receive the same fine-tuning.
The clarification about scope also makes this paper's contribution too small for ICLR. If you are saying you just compare the SAM-MedSAM pair, what is the impact and the knowledge you want to bring in broad venue (i guess most audience reading ICLR are not in medical field and they want some more broad useful/applicable findings?) Even in medical field, these two are back to 2023 models.
Thus, i still find the contribution limited and paper needs more time to polish, and the response does not give me sufficient evidence to change my assessment.
Dear Reviewer,
First, we would like to thank you again for your comments and follow-up. We take your feedback seriously and are willing to refine the paper accordingly.
In both your review and follow-up, you suggested that the selected polyp segmentation task is not sufficiently challenging and that medical AI research offers a narrow contribution to the ICLR audience, raising concerns about the value of our work. We disagree with this assessment. We are particularly concerned about your use of “guess” when suggesting that most of the ICLR audience is outside the medical domain. We view this statement as reflecting an explicit bias against medical AI, potentially amounting to discrimination. The ICLR community includes many excellent medical AI papers and readers working in this field, ourselves included. We therefore do not consider our exploration of pruning in the medical domain to be inherently too narrow a contribution to ICLR.
We would also like to highlight that the ICLR review guidelines ask reviewers to “be open-minded” when considering work beyond their own areas of expertise. Your repeated requests for broader validation to other non-medical or more challenging medical datasets suggest to us that you may be less familiar with this area. As stated in the ICLR 2027 guidelines, “Reviewing a submission: step-by-step,” the assessment should focus on the substantive issues that determine a paper’s merit, rather than additional experiments or typos that can add to everyone’s workload. Maybe that is a good tips for us!
We nevertheless take your assessment that the paper appears rushed seriously (we guess you haven’t looked closely at the mathematics behind structured pruning; perhaps taking a closer look at our pruning method, with some help from AI, might lead you to a different understanding) and appreciate your feedback on its readability. We of course will continue to refine the paper regardless of the ICLR outcome (which may now matter less to us nowadays). We welcome further discussion and exchange with you! And sincerely appreciate your contribution to ICLR and look forward to substantive, high-quality feedback in the future!
Sincerely,
The Authors
Generalization beyond one SAM-MedSAM ViT-B pair
This is a valid limitation. MedCore relies on parameter-aligned SAM and MedSAM checkpoints so that reset-to-SAM is meaningful. The paper tests several target datasets and modalities (polyp, BUSI, ISIC2018, ACDC), but all use the official SAM ViT-B to MedSAM ViT-B pair. Thus the experiments support cross-dataset and cross-modality behavior for this backbone pair, not a claim that the same reset score will necessarily work for other backbones or adaptation paths. Appendix H states this assumption explicitly. For architectures that change parameterization, the reset intervention would need an alignment or distillation-based analogue.
Pruning baselines and final fine-tuning
"Do the final gains over pruning baselines remain after the same fine-tuning?" We did not run terminal fine-tuning for every Appendix G baseline. The reason is methodological: Appendix G asks whether the score selects better structures before terminal recovery, so it disables final fine-tuning for all methods, including MedCore. Under that matched setting, MedCore leads 15/16 metric-budget cells. We agree that a second study where all pruning baselines receive identical terminal fine-tuning would test whether those score differences persist after recovery. That experiment is not in the submitted paper, so we should not claim it.
Boundary accuracy and dataset difficulty
The concern about Figure 2 is fair. The qualitative example appears to show a relatively regular polyp, and the caption does not identify the dataset. The quantitative evidence is broader than that figure: Table 1 macro-averages CVC-ClinicDB, CVC-ColonDB, and Kvasir-SEG; Table 6 includes BUSI and ISIC2018; ACDC is reported per cardiac structure in Table 2. The results are not uniformly positive: BUSI degrades under aggressive compression, and ACDC BF1/HD95 also drop for MedCore h70_m95. We view these as useful stress cases showing that MedCore is a compression framework requiring target-domain validation, not a universal boundary improvement method. Figure 2 should have used a harder case and named the dataset.
Presentation and inconsistent numbers
We agree. The inconsistent precision in Table 4, the h50_m70 FLOPs mismatch (385.2G versus 377.0G), and unclear configuration names reduce trust in the paper. The intended naming is hX_mY, where h is the head sparsity target and m is the MLP sparsity target; sequential variants add a second conservative pass. The FLOPs mismatch should be corrected or clearly tied to the accounting context. These are presentation errors rather than changes to the pruning method, but they should be fixed.
Distillation-based compact students
We did not compare to a compact student distilled from MedSAM under matched training and inference budgets. Table 1 includes EfficientSAM, SlimSAM, QMedSAM, and medical segmentation backbones, but these are not a controlled student-distillation baseline. The paper's narrower contribution is direct structured pruning of an already adapted MedSAM checkpoint while preserving boundary-sensitive structures. A matched student-distillation comparison would be valuable future work and may be stronger at very small FLOP budgets.
Reset score for another backbone pair
We cannot claim this from current experiments. The reset score should be most meaningful when the two checkpoints are parameter-aligned and the adaptation path preserves architecture, as with SAM ViT-B to MedSAM ViT-B. For another aligned pair, the method is directly applicable, but its usefulness would need validation. For non-aligned backbones, reset-to-SAM is not well-defined without an additional mapping.