acceptodds
Withdrawn from ICLR 2027

MedCore: Boundary-Preserving Medical Core Pruning for MedSAM

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical segmentation foundation models such as SAM and MedSAM provide strong prompt-driven segmentation, but their image encoders are still too large for many clinical settings. Compression is also risky in medicine because a model can keep high Dice while losing boundary fidelity. We propose MedCore, a structured pruning framework for MedSAM. The main idea is to preserve two kinds of structures: structures that became important during SAM-to-MedSAM adaptation, and structures that have high boundary leverage. We identify the first type by a dual-intervention score comparing zeroing a group with resetting it to its original SAM weights. We identify the second type by boundary-aware Fisher estimation. We also introduce a boundary leverage principle, which relates first-order boundary displacement to boundary logit perturbation divided by the spatial gradient norm. This principle explains why boundary metrics can degrade even when Dice remains high. On polyp segmentation benchmarks, MedCore reduces parameters by 60.0% and FLOPs by 58.4% while achieving Dice 0.9549, Boundary F1 0.6388, and HD95 5.14 after post-pruning fine-tuning. It also reaches 86.6% parameter reduction and 90.4G FLOPs, retaining Dice and HD95 near the fine-tuned MedSAM reference, with lower BF1. Our analysis shows that MedSAM lies in a head-fragile boundary regime: the median 95th-percentile boundary leverage is 2.887 times as high for head-pruning as for MLP-pruning steps, consistent with BF1 and HD95 degradation. Our code is available at https://anonymous.4open.science/r/MedCore-8461.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical image segmentation, MedSAM, structured pruning, boundary leverage principle

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RFMbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MedCore, a structured pruning method to reduce MedSAM’s model size while preserving segmentation boundaries. It selects attention heads and MLP groups to prune by estimating the effect of removing them or resetting their weights to the original SAM checkpoint. Thet also propose boundary-weighted Fisher scores to guide prune, followed by recovery training and continual fine-tuning. Experiments show that they can achieve a much compressed model with almost no change on DICE.

Critical strengths
  1. The motivation is clear: they use pruning methods to preserve model performance while compressing model sizes.
  2. Their proposed boundary leverage part is interesting; I think I didn't see similar work before.
  3. Their matched pruning comparisons in the appendix and the component ablations provide useful evidence, particularly for the boundary-aware Fisher scoring.
Critical weaknesses
  1. All experiments use one SAM-MedSAM ViT-B checkpoint pair. It is unclear whether these findings are medical image tasks generalizable or just this particular model pairing.
  2. The strongest pruning methods are in appdenix and don't include final fine-tuning. It is less convincing to the advantage over existing methods bc the main tables in paper are just MedSAM vs. compressed model.
  3. I think the boundary scoring idea is interesting, but I feel the datasets the authors choose are less challenging in this task. Such as the visualization of the results in Figure 2 (btw, the authors don't explain which dataset it is in the figure title), it is a pretty regular round object; thus, it is hard to show the benefit of boundary accuracy. I think the results can be more convincing if the authors pick a more challenging dataset, whose boundary accuracy do matter.
  4. My major concern also includes the paper presentation. I feel the entire paper looks rushed, like inconsistent digits (4 digits for some metrics and even 6 digits for some others in Table 4); also inconsistent numbers reported (for example, 385.2 G vs. 377.9 G for h50_m70). Also, the h50_m70 is also like an intermediate config version of the model, which is hard to follow and understand during reading. I suggest the authors could spend more time polishing their writing and presentations.
Questions for authors
  1. Does the reset score remain useful for another backbone pair or backbone config?
  2. Do the final fains over the pruning baselines in Table 7 remain after the same fine-tuning?
  3. How does MedCore compare with a compact student distilled from MedSam under comparable training and inference budgets?
4 comments

Generalization beyond one SAM-MedSAM ViT-B pair

This is a valid limitation. MedCore relies on parameter-aligned SAM and MedSAM checkpoints so that reset-to-SAM is meaningful. The paper tests several target datasets and modalities (polyp, BUSI, ISIC2018, ACDC), but all use the official SAM ViT-B to MedSAM ViT-B pair. Thus the experiments support cross-dataset and cross-modality behavior for this backbone pair, not a claim that the same reset score will necessarily work for other backbones or adaptation paths. Appendix H states this assumption explicitly. For architectures that change parameterization, the reset intervention would need an alignment or distillation-based analogue.

Pruning baselines and final fine-tuning

"Do the final gains over pruning baselines remain after the same fine-tuning?" We did not run terminal fine-tuning for every Appendix G baseline. The reason is methodological: Appendix G asks whether the score selects better structures before terminal recovery, so it disables final fine-tuning for all methods, including MedCore. Under that matched setting, MedCore leads 15/16 metric-budget cells. We agree that a second study where all pruning baselines receive identical terminal fine-tuning would test whether those score differences persist after recovery. That experiment is not in the submitted paper, so we should not claim it.

Boundary accuracy and dataset difficulty

The concern about Figure 2 is fair. The qualitative example appears to show a relatively regular polyp, and the caption does not identify the dataset. The quantitative evidence is broader than that figure: Table 1 macro-averages CVC-ClinicDB, CVC-ColonDB, and Kvasir-SEG; Table 6 includes BUSI and ISIC2018; ACDC is reported per cardiac structure in Table 2. The results are not uniformly positive: BUSI degrades under aggressive compression, and ACDC BF1/HD95 also drop for MedCore h70_m95. We view these as useful stress cases showing that MedCore is a compression framework requiring target-domain validation, not a universal boundary improvement method. Figure 2 should have used a harder case and named the dataset.

Presentation and inconsistent numbers

We agree. The inconsistent precision in Table 4, the h50_m70 FLOPs mismatch (385.2G versus 377.0G), and unclear configuration names reduce trust in the paper. The intended naming is hX_mY, where h is the head sparsity target and m is the MLP sparsity target; sequential variants add a second conservative pass. The FLOPs mismatch should be corrected or clearly tied to the accounting context. These are presentation errors rather than changes to the pruning method, but they should be fixed.

Distillation-based compact students

We did not compare to a compact student distilled from MedSAM under matched training and inference budgets. Table 1 includes EfficientSAM, SlimSAM, QMedSAM, and medical segmentation backbones, but these are not a controlled student-distillation baseline. The paper's narrower contribution is direct structured pruning of an already adapted MedSAM checkpoint while preserving boundary-sensitive structures. A matched student-distillation comparison would be valuable future work and may be stronger at very small FLOP budgets.

Reset score for another backbone pair

We cannot claim this from current experiments. The reset score should be most meaningful when the two checkpoints are parameter-aligned and the adaptation path preserves architecture, as with SAM ViT-B to MedSAM ViT-B. For another aligned pair, the method is directly applicable, but its usefulness would need validation. For non-aligned backbones, reset-to-SAM is not well-defined without an additional mapping.

Dear Reviewer,

Thank you again for your constructive feedback. As the discussion period closes on November 18, we remain available to provide any further clarification or address any remaining questions.

If you feel that our rebuttal has adequately addressed your concerns, we would greatly appreciate your reconsideration of the overall assessment.

Sincerely,

The Authors

Reviewer_RFMbOpenReview ↗

Authors try to answer my questions but not solve any.

Thank you for the response and for clarifying the scope of this work.

However, my main concerns remain unresolved. the response acknowledges the issues i raised, but provide no concrete corrections or additional results (i just checked the latest paper pdf). For example, the inconsistent FLOPs values. the qualitativ e concern could also be addressed by finding the dataset and providing a more challenging example that illustrates boundary preservation.

These seem feasible to address directly, yet the rebuttal did any. I really wish the authors can spend more time on their paper, also think about ur response; (this also reflect my impression when reading this paper, feel very rushed and should spend more time to polish before submission).

I understand the rational for comparing pruning scores without fine-tuning. However, this does not answer whether MedCore's advantage remains when all pruning baselines receive the same fine-tuning.

The clarification about scope also makes this paper's contribution too small for ICLR. If you are saying you just compare the SAM-MedSAM pair, what is the impact and the knowledge you want to bring in broad venue (i guess most audience reading ICLR are not in medical field and they want some more broad useful/applicable findings?) Even in medical field, these two are back to 2023 models.

Thus, i still find the contribution limited and paper needs more time to polish, and the response does not give me sufficient evidence to change my assessment.

Dear Reviewer,

First, we would like to thank you again for your comments and follow-up. We take your feedback seriously and are willing to refine the paper accordingly.

In both your review and follow-up, you suggested that the selected polyp segmentation task is not sufficiently challenging and that medical AI research offers a narrow contribution to the ICLR audience, raising concerns about the value of our work. We disagree with this assessment. We are particularly concerned about your use of “guess” when suggesting that most of the ICLR audience is outside the medical domain. We view this statement as reflecting an explicit bias against medical AI, potentially amounting to discrimination. The ICLR community includes many excellent medical AI papers and readers working in this field, ourselves included. We therefore do not consider our exploration of pruning in the medical domain to be inherently too narrow a contribution to ICLR.

We would also like to highlight that the ICLR review guidelines ask reviewers to “be open-minded” when considering work beyond their own areas of expertise. Your repeated requests for broader validation to other non-medical or more challenging medical datasets suggest to us that you may be less familiar with this area. As stated in the ICLR 2027 guidelines, “Reviewing a submission: step-by-step,” the assessment should focus on the substantive issues that determine a paper’s merit, rather than additional experiments or typos that can add to everyone’s workload. Maybe that is a good tips for us!

We nevertheless take your assessment that the paper appears rushed seriously (we guess you haven’t looked closely at the mathematics behind structured pruning; perhaps taking a closer look at our pruning method, with some help from AI, might lead you to a different understanding) and appreciate your feedback on its readability. We of course will continue to refine the paper regardless of the ICLR outcome (which may now matter less to us nowadays). We welcome further discussion and exchange with you! And sincerely appreciate your contribution to ICLR and look forward to substantive, high-quality feedback in the future!

Sincerely,

The Authors

Reviewer_Eh9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MedCore, a structured pruning method for the MedSAM image encoder. It prunes attention heads first and then MLP groups. Each group is scored by two interventions. One zeroes the group and the other resets it to the original SAM weights. Both are approximated with a boundary weighted Fisher. The paper also gives a boundary leverage argument linking boundary shift to logit change divided by logit gradient norm. From this it claims heads hurt boundaries more than MLPs. On polyp data the method removes up to 86.6 percent of parameters and stays close to fine-tuned MedSAM on Dice and HD95. It also reports BUSI and ISIC and ACDC results and ablations and a comparison with eight pruning criteria.

Critical strengths
  1. I like the matched comparison in Appendix G. Eight criteria run on the same encoder with the same groups and calibration data and MedCore wins 15 of 16 cells. The BF1 margin also grows with compression. This shows the scoring itself matters and not just an architecture swap. It also covers the Group Fisher and Hessian-aware baselines that the attached draft review calls missing.
  2. The problem is well chosen. Boundary quality is what matters for lesion measurement and a model can keep high Dice while its contours drift. I also think reporting BF1 and HD95 next to Dice is the right call for compression work in medical imaging.
  3. The compressed model is practically useful. The 12.1M parameter version keeps Dice and HD95 close to fine-tuned MedSAM. The idea of using the SAM checkpoint as a reference to find what changed during adaptation is simple and sensible.
Critical weaknesses
  1. The attribution to the proposed scoring is confounded. The criteria comparison has no fine-tuning but every headline number has it. After fine-tuning the smallest model beats larger ones on BF1 so fine-tuning seems to dominate.
  2. Table 2 looks like fine-tuned MedCore against a MedSAM that was not fine-tuned since the ClinicDB Dice of 0.9264 matches the zero-shot level. I think the ISIC BF1 jump from 0.24 to 0.48 is then mostly fine-tuning and not pruning.
  3. The ablation does not support the dual intervention story. Removing reset or variance raises BF1 by 0.048 and 0.067 at h70_m70 and removing variance changes nothing at h84_m95. Everything is one seed with no error bars.
  4. The boundary leverage claims go beyond the evidence. The theorem is a standard level set result and it is not used in the actual score. The region leverage ratio of 3.06 is above the boundary ratio of 2.89 so head pruning is not boundary specific. The law rests on only 15 paired cells that are not independent and the first-order assumption is stretched by pruning up to 84 percent.
  5. Section 5.3 shows h50_m70 beating h30_m80 at equal size and this cuts against the advice to protect heads.
Questions for authors
  1. Is MedSAM in Tables 2 and 6 fine-tuned with the same protocol as MedCore? If not please add that baseline.
  2. Were the Table 1 baselines trained on the same splits? SAMed at Dice 0.74 looks undertrained.
  3. Did the Appendix G baselines get the same block budgets and protected blocks as MedCore?
  4. FLOPs for h70_m95 are 245.0G in Table 1 and 240.9G in Table 2. Which is right?
2 comments

Was MedSAM fine-tuned with the same protocol?

Yes for Table 1: the caption states "MedSAM + fine-tuning and MedCore use the same fine-tuning protocol." The full-model reference there is MedSAM + fine-tuning, with Dice 0.9578, BF1 0.6809, and HD95 5.11. MedCore h84_m95 reaches Dice 0.9550, BF1 0.6462, and HD95 5.12 at 12.1M parameters and 90.4G FLOPs. Thus the headline polyp comparison is not against a zero-shot MedSAM.

Table 2 is less clear. It labels the baseline as "MedSAM" and reports ClinicDB Dice 0.9264, matching the unfine-tuned row in Table 1 rather than MedSAM + fine-tuning. Your reading is therefore reasonable. The cross-modality table should not be used to claim that pruning alone beats a same-protocol fine-tuned full model on ISIC/ClinicDB. Its proper interpretation is a target-dataset evaluation showing that pruned MedCore after target fine-tuning can improve over the provided MedSAM baseline on some domains, while BUSI and ACDC show limits. The stronger same-protocol evidence is Table 1 for polyp macro results.

Does fine-tuning dominate the gains?

Fine-tuning is important, but it does not explain the matched pruning-score result. Appendix G disables terminal post-pruning fine-tuning for every criterion, including MedCore. In that setting MedCore still obtains the best BF1 at all four compression levels: 0.5162, 0.4078, 0.3144, and 0.2539. It also improves over Zero-only Fisher, Group Fisher, and Hessian-aware ViT under the same masks/groups/budgets/calibration protocol. Final fine-tuning then recovers usable performance, but the pruning score determines which compact subnetwork is being recovered.

Ablations and the dual-intervention story

We agree that Table 3 does not show every component is monotonically helpful. It supports a more limited claim: boundary-aware Fisher is consistently important, especially at h70_m95 where BF1 drops by 0.127 without it; reset helps under aggressive compression, dropping BF1 by 0.082 at h70_m95 and 0.018 at sequential h84_m95; variance is helpful at h70_m95 but neutral at sequential h84_m95. At h70_m70, removing reset or variance improves BF1, so the text should say these terms are safeguards whose value depends on compression level and dataset mixture, not universally beneficial components. We also agree that single-seed results without error bars are a limitation.

Boundary leverage and head pruning

We do not intend to claim head pruning is boundary-specific in the sense of affecting only boundaries. Appendix D.2 explicitly states BSR is 0.942 and says, "we do not claim that head pruning is boundary-specific. We claim that head pruning is high-boundary-leverage." That is the needed statement for boundary fragility: Table 4 shows head steps have median λ95bd=3.961λ_{95}^{bd}=3.961 versus 1.372 for MLP steps, with 15/15 paired wins, and metric damage densities are also higher for heads (BF1 ratio 2.607, HD95 ratio 2.432). The theorem is a local diagnostic, not the pruning score itself; it motivates why BF1/HD95 can degrade even when Dice remains high.

h50_m70 versus h30_m80

This does not contradict conservative head pruning. h30_m80 has less head pruning but a different MLP allocation and falls on the min-retention/non-monotonic region shown in Figure 3. Section 5.3 uses exactly this example to argue that boundary performance depends on structural allocation, block constraints, and surviving groups, not just a scalar "more heads protected" rule. The practical rule is to use calibration-based boundary leverage to guide the head/MLP balance.

Baseline training, Appendix G budgets, and FLOPs

Table 1 compares against published efficient/medical models with different pipelines, so SAMed may indeed be disadvantaged relative to a fully retuned implementation; that is why Appendix G is the controlled pruning comparison. Appendix G baselines share the same MedSAM ViT-B encoder, structured groups, calibration data, target sparsity levels, metrics, and 100-step stabilization. The FLOPs discrepancy is a presentation issue: h70_m95 is 245.0G in Table 1 and 240.9G in Tables 2/6; this should have been harmonized or footnoted.

Dear Reviewer,

Thank you again for your constructive feedback. As the discussion period closes on November 18, we remain available to provide any further clarification or address any remaining questions.

If you feel that our rebuttal has adequately addressed your concerns, we would greatly appreciate your reconsideration of the overall assessment.

Sincerely,

The Authors

Reviewer_FrgAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MedCore prunes attention heads and MLP groups from MedSAM while using the original SAM checkpoint as an adaptation reference. Its importance score combines estimated loss changes from zeroing a group and resetting that group to its SAM weights. The estimates use boundary-weighted Fisher information, a geometric mean of SAM and MedSAM Fisher values, and aggregation across calibration datasets. Block-specific budgets and a head-to-MLP cascade are followed by recovery and target-dataset fine-tuning. A local boundary-displacement analysis motivates the emphasis on boundary fidelity. Experiments report region and boundary metrics on polyp datasets and additional medical modalities.

Critical strengths
  1. The reported compression preserves much of the fine-tuned model's accuracy.
  2. Appendix G provides a useful controlled pruning comparison. Eight scoring baselines use the same encoder, structured groups, calibration data, and 100-step inter-stage stabilization. MedCore obtains the highest Dice, IoU, and BF1 at all four reported compression levels.
Critical weaknesses
  1. The adaptation score lacks direct validation. Boundary-weighted Fisher approximates Dice-plus-BCE intervention costs without justifying the omitted first-order term or Cross-Fisher curvature. Neither measured intervention costs nor a MedSAM-only Fisher reset baseline validates this choice. At h70_m70, removing reset or variance improves BF1, weakening the evidence for these components.
  2. Appendix G omits terminal fine-tuning for pruning baselines, leaving MedCore’s final advantage under matched training unestablished. The unexplained BF1 gap between “w/o reset” (0.3628) and “Zero-only Fisher” (0.2784) at h70_m70 also requires reconciliation. The target fine-tuning schedule, block mixture weights, and BF1 matching tolerance remain unspecified.
Questions for authors

See weakness.

2 comments

Fine-tuning and the pruning-score attribution

"Appendix G omits terminal fine-tuning for pruning baselines." Correct: this was intentional, not an omission. Appendix G disables terminal post-pruning fine-tuning for all methods, including MedCore, to isolate the importance score. Under this controlled setting, all methods share the same encoder, groups, calibration data, target sparsities, metrics, and 100-step inter-stage stabilization. MedCore leads 15/16 metric-budget cells in Table 7, and its BF1 advantage over the strongest competitor increases with compression. Table 1 answers a different question: after a compact mask is selected, can the resulting model be made useful under the same target fine-tuning protocol as the full MedSAM reference? The answer is yes: h84_m95 has 12.1M parameters and 90.4G FLOPs while remaining within 0.0028 Dice and 0.01 HD95 of fine-tuned MedSAM, with a BF1 trade-off.

Direct validation of the dual-intervention score

"The adaptation score lacks direct validation." We agree the exact intervention costs are not exhaustively measured for every group; the scoring is a tractable Fisher approximation. The direct evidence we provide is comparative and component-level: Table 7 compares against Zero-only Fisher, Vanilla Fisher, Group Fisher, and Hessian-aware ViT under matched conditions. This tests whether the full boundary- and adaptation-aware score improves decisions beyond standard sensitivity criteria. Component ablation in Table 3 is mixed at moderate settings but supports reset under aggressive compression: removing reset lowers BF1 by 0.082 at h70_m95 and by 0.018 in the sequential h84_m95 setting. We should not have implied that reset is uniformly beneficial at every sparsity, since h70_m70 is a counterexample.

First-order term, Cross-Fisher, and the reset baseline

The first-order term is omitted because the Fisher score follows the standard second-order pruning approximation around a trained checkpoint, where the empirical gradient is treated as small. This is an approximation, not an exact identity, and it can fail when pruning is very aggressive. Cross-Fisher is used only for the reset intervention: Eq. (11) takes the geometric mean of SAM and MedSAM Fisher so that a large parameter displacement contributes only when it is functionally sensitive at both endpoints. A MedSAM-only reset-Fisher baseline is not separately reported. The closest reported controls are w/o reset, Zero-only Fisher, Vanilla Fisher, and Original Medical Core; these do not fully isolate that variant, so we acknowledge this as a missing ablation rather than claim it is resolved.

BF1 gap between "w/o reset" and "Zero-only Fisher"

These rows are not identical experiments. "w/o reset" in Table 3 is an internal component ablation of the full MedCore pipeline, retaining other choices such as boundary-aware scoring, distribution aggregation, and block allocation. "Zero-only Fisher" in Table 7 is a matched external pruning criterion using only the zero intervention as defined for that baseline. Thus, the BF1 difference at h70_m70 reflects more than reset alone; it includes the remaining MedCore allocation and boundary-aware machinery. The paper should have made this distinction clearer.

Missing protocol details

The target fine-tuning schedule and hyperparameters are stated to be in Appendix E, while Appendix D specifies calibration size, recovery steps, learning rate, and recovery-loss weights. The BF1 matching tolerance and block mixture weights are not sufficiently explicit in the submitted text. We cannot add new experiments in the rebuttal, but we agree that these are reproducibility-critical details and will include them in the camera-ready and code release.

Dear Reviewer,

Thank you again for your constructive feedback. As the discussion period closes on November 18, we remain available to provide any further clarification or address any remaining questions.

If you feel that our rebuttal has adequately addressed your concerns, we would greatly appreciate your reconsideration of the overall assessment.

Sincerely,

The Authors

Comments on the paper

1 comment

We thank the reviewers for the careful feedback. We are encouraged that the matched pruning comparison in Appendix G was viewed as useful (FrgA, Eh9f, RFMb), that the boundary-quality motivation was recognized (Eh9f, RFMb), and that the compressed MedSAM model was seen as practically useful (Eh9f). The main point we want to clarify is the separation between (i) evidence that the pruning score is better under matched no-terminal-fine-tuning conditions, and (ii) evidence that the resulting compact model remains useful after the same post-pruning fine-tuning protocol used for the full MedSAM reference.

The fine-tuning confound is a fair concern. The paper already includes both views: Table 7 disables terminal fine-tuning for all pruning criteria, including MedCore, and therefore isolates the score; Table 1 compares final deployed models after post-pruning fine-tuning, with the full MedSAM reference fine-tuned by the same protocol. Table 7 is the key evidence for attribution to pruning: MedCore is best in Dice, IoU, and BF1 at all four compression levels and best in HD95 in three of four, leading 15/16 metric-budget cells. Its BF1 margin over the strongest matched pruning baseline grows from 0.0118 at h40_m30 to 0.0467 at h84_m95, which is hard to explain by final fine-tuning because final fine-tuning is disabled there.

Several reviewers also noted presentation issues and numerical inconsistencies. We agree these hurt readability. Within the rebuttal, the intended reading is: Table 1 macro-average h50_m70 FLOPs should be aligned with the per-dataset accounting in Table 6 where appropriate, and h70_m95 is 245.0G in the macro table versus 240.9G in the single-dataset/cross-modality table because those rows report different evaluation/accounting contexts. These should have been stated explicitly and will be cleaned in the camera-ready.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.