acceptodds
Under review as a conference paper at ICLR 2027

Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models

12 views

Abstract

Vision foundation models (VFMs) provide transferable visual representations, but their inference cost limits deployment in many settings. Fine-tuning specializes a VFM without shrinking it; distillation shrinks it, but through expensive teacher-guided training. Structured pruning offers a more direct route by extracting a compact subnetwork from the VFM. However, as compression gets aggressive, existing methods fail because they treat subnetwork selection and recovery as separate axes. Across layers, capacity must follow task sensitivity rather than spread uniformly. Within a layer, a removal cannot be judged by importance alone; its impact also depends on how well the lost contribution can be recovered from what remains. Selection and recovery must therefore be decided jointly. We introduce Carve VFM, which resolves this interdependence. Given a VFM, a calibration set, and a parameter budget, it (i) searches for task-specific capacity allocations across layers; (ii) guides the search via an adaptive recovery-aware fitness criterion and (iii) reconstructs removed behavior through iterative closed-form low-rank corrections merged into the surviving weights. The result is a dense task model produced in minutes on a single GPU, without training. Across twelve benchmarks and four models from DINOv3 and MetaCLIP-2, Carve VFM ranks first among all evaluated structured-pruning methods at every budget, averaging +9.2%; crucially, this margin widens from +3.0% at 35% parameter removal to +16.3% at 70%. Further fine-tuning surpasses the strongest evaluated edge model 88.1% vs. 83.4%, and carved models cut compute and memory while delivering 1.3-2.0x higher GPU throughput than the uncompressed one. These results show that the compact model you need is inside the VFM; it only has to be carved out.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision foundation models, model compression, recovery-aware pruning, training-free compression

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_8ikhOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper aims to get a small, task-specific dense model out of a big VFM, given a labeled calibration set (512 images) and a parameter budget BB. Carve VFM works in three steps:

  • It first caches OBC-style greedy removal curves for every layer and operator: MLP channels, whole heads (H), or within-head coordinates (C).
  • It then runs an evolutionary search over per-layer MLP/attention retention (slmlp,slatt)(s_l^{\mathrm{mlp}}, s_l^{\mathrm{att}}) and the operator ol∈{H,C}o_l \in \{\mathrm{H}, \mathrm{C}\}. The search is scored by a cheap additive "recoverable damage" proxy, and falls back to held-out linear-probe accuracy when an audit says the proxy's ranking can't be trusted.
  • Finally, it carves the chosen architecture and folds a ridge + SVD-truncated correction ΔWr\Delta W_r into the surviving weights, so nothing extra remains at inference.

Evaluation covers 4 backbones (DINOv3 S/B/L, MetaCLIP-2 B), 12 classification tasks and 3 budgets each, mostly with frozen-backbone linear probing. There is also a full fine-tuning comparison against compact ImageNet models and CPU/GPU throughput measurements on ViT-L.

Critical strengths

1. What comes out is a plain dense model, and the speedups are real. No masks, no sparse kernels, no extra low-rank branches. That's why Table 11 can show actual wall-clock gains (1.31.3–2.0×2.0\times GPU, 1.91.9–3.2×3.2\times CPU throughput vs. full ViT-L) instead of theoretical FLOP savings. I also admitted that the authors didn't hide the awkward part: batch-1 GPU latency actually goes up slightly.

2. The experiments are broad and reported in a way I could actually dig into. 4 backbones ×\times 12 tasks ×\times 3 budgets, three seeds with stds, per-dataset tables, wall-clock costs, and a limitations section that is fairly upfront.

Critical weaknesses

1. It is unclear whether carving a larger VFM is preferable to using a smaller pretrained model from the same family. Table 1 shows that DINOv3 ViT-B carved to 20M achieves 54.3% mean accuracy, while the original ViT-S (21.6M) achieves 87.2%. Similarly, carving ViT-L to 100M gives 83.7%, below the original ViT-B (85.7M) at 90.6%. Although the budgets are not exactly matched in the first comparison, these results raise an important practical question: why carve a larger model when a smaller pretrained model may already perform substantially better? I would like to see a more direct comparison within the same pretrained model family.

2. It is hard to tell which component actually drives the improvements. In the decoupling experiment, adding repair to uniform pruning increases mean accuracy from 31.7% to 78.2%, while the full method reaches 85.4%. This suggests that recovery accounts for much of the improvement. Meanwhile, the adaptive search switches to global guidance in 79% of MetaCLIP-2 runs and 89% of ViT-L runs. How much does the recoverability-aware local fitness really contribute on these larger models?

Questions for authors

1. How are checkpoints selected? Probes and fine-tuning are evaluated at epochs 1/3/5/10, and Table 10 reports the "best checkpoint." Best according to which split? The search stage explicitly avoids the test set, but I couldn't find the same statement for downstream checkpoint selection. If it were test accuracy, that would be a real problem for the whole evaluation, so a clear statement (and ideally fixed-epoch results, e.g., epoch 10) would settle this. This one could change my score.

2. Is CORP's DTD number expected? At the mild budgets, CORP gets 57.9–63.8% on DTD across all four backbones, while Carve and SVD-LLM are around 75–83% on the same settings. On ViT-S at 14M, DTD alone accounts for roughly half of Carve's 2.7-point margin over CORP. Could you double-check this configuration? More generally, were the baselines run with their original recovery/compensation steps? Several (e.g., SNIP–Mag., SnapViT) end up at or below Random, which surprised me.

3. What exactly is removed in the "without repair" setting? The decoupling paragraph reports 28.2% for FLAP subnetworks without repair at 14M. But Table 1's FLAP on the same three datasets (C100, CUB, BloodMNIST) averages 57.2%. Also at 14M, uniform without repair (31.7%) is below Random on those datasets (40.6%, from Table 3), although at 6M it is above (31.5% vs. 27.6%). Is FLAP's own bias compensation stripped here? Does "uniform without repair" still use the OBC removal order? I need this to judge the repair-vs-search split in Weakness (2).

3 comments

We thank the reviewer for the careful reading, and for noting the deployment value and the transparency of the paper.

Q1. Checkpoint selection. Details are in our reply to Reviewer BxTr. For every method, the checkpoint is chosen among epochs {1, 3, 5, 10} on a validation subset of 20% of the training split; the test split is never used for selection. The paper's wording ("full training set") omitted this and will be corrected. The only difference is how the subset is drawn: the original script re-sampled it every epoch, while SliceGPT-ViT and OSSCAR-ViT, run later, used a revised script that keeps one stratified subset fixed. To remove this difference, all methods are now reported at the fixed final epoch (epoch 10), as the reviewer suggested. Epoch 10 was already the chosen checkpoint in 81–94% of runs, 12-task means change by 0.00–0.23 pp, and all eight rankings are unchanged (table in our reply to Reviewer BxTr). For full fine-tuning (Table 11) at epoch 10, the 14M specialist reaches 87.5% vs. 83.4% for the strongest native compact model (+4.1 pp).

W1. Carving vs. smaller pretrained models. See our reply to Reviewer YPy6 (Tables R1–R3). When a distilled model of the target size exists, it is preferable at aggressive budgets. Those models were distilled from a 7B teacher on ~1.7B images, while Carve VFM uses 512 task images and minutes on one GPU; a short task fine-tuning closes most of the gap. It is most useful where no suitable distilled model exists: single-size domain models (RAD-DINO → 35M beats ViT-S+ by 7–10 pp), systems built on the teacher (the teacher's linear head transfers unchanged at 81.8% vs. 4.1% for native ViT-S), and large-model gaps (Qwen2.5-VL 7B → 5.5B beats the native 3B on all 7 tasks).

W2. Which component drives the gains? The search is repair-aware: every candidate is scored by its accuracy after repair, so architectures are selected for how well repair can recover them. Repair therefore looks like the main driver, but much of its benefit comes from an architecture chosen for it. Holding repair fixed isolates the search: a FLAP-selected architecture reaches 63.5%, a uniform one 78.2%, and the repair-aware search 84.9% (ViT-S, 14M; CIFAR-100, CUB-200, BloodMNIST; seed 17). That is +6.7 pp from search alone. (The paper's +7.2 pp used the three-seed Carve mean; we now use the matching seed.)

On the fitness: the local OBS score ranks the population cheaply (32 candidate builds instead of 120–200), and held-out accuracy audits the top 12, overriding the ranking when they disagree. The high switch rate on ViT-L (89%) and MetaCLIP-2 (79%) shows why the audit is needed on larger models. On ViT-S, local-only loses 5.8 pp at 6M and the adaptive audit recovers 4.3 pp.

Q2. CORP on DTD. The CORP rows are our re-implementation of the CORP recipe (Wanda-scored MLP channel pruning and QK-dimension pruning with closed-form logit compensation and bias correction), because no official code was available at submission time. The authors added a repository link in their latest version (9 Nov 2026, after our submission). Its DTD accuracy is low on all four backbones at the mild budget (57.9–63.8%), so this is systematic and not a single-run artifact; we will run it with the new available official code and see whether it is specific to our re-implementation.

Recovery steps. FLAP and CORP keep their own compensation. Random, SNIP–Mag. and SnapViT use the same uniform per-layer MLP sparsity and bias correction, and differ only in which attention heads are removed under a global head budget.

Baselines at or below Random. We attribute the gap to how head removal is distributed. SNIP–Mag. and SnapViT leave at least one layer with a single head (of six) in every run (24/24 at ViT-S 14M), concentrating removal in a few layers. Random keeps at least 3 heads in every layer in half of the runs.

Q3. What "without repair" removes. We will add these definitions to the appendix. All variants share calibration data, budget projection and parameter budget.

  • Uniform −R / +R: a uniform per-layer fraction with Carve VFM's OBS removal order. −R takes the pruned model directly, with no compensation; +R applies Carve VFM's closed-form repair.
  • FLAP −R / +R: FLAP's masks, with FLAP's own bias compensation replaced by none (−R) or Carve VFM's repair (+R). FLAP in Table 4 (57.2% on these tasks) includes its bias compensation; FLAP −R (28.2%) does not.
  • Uniform −R vs. Random (31.7% vs. 40.6% at 14M): Random (Table 4) includes bias correction, while uniform −R has no compensation. In addition, OBS selects channels that are cheap to remove given its compensating update; without that update the ordering carries no guarantee. We do not yet have an explanation for why the gap reverses at 6M (31.5% vs. 27.6%).

The authors have satisfactorily addressed my concerns in their rebuttal. I appreciate their clarifications and additional explanations, which have resolved my remaining questions. I am satisfied with the response and will update my recommendation to Accept.

We thank the reviewer for the discussion and for raising the score. The revised version includes these clarifications.

Reviewer_BxTrOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Carve VFM, a method for creating compact, task-specific vision models. It addresses a gap in pruning methods that select components without adequately considering weight recovery. The method uses a small labelled calibration set to choose attention and feed-forward configurations. It then reconstructs the retained weights to limit accuracy loss. Validation covers multiple vision backbones, classification datasets and compression budgets. It includes baseline comparisons, component ablations and hardware measurements. The compressed models often retain accuracy close to the full models. They also achieve higher GPU throughput and faster CPU execution. However, accuracy gains over competing methods are not consistent across all settings. This is a promising practical compression framework with substantial empirical coverage. Its strongest contribution is integrating architecture selection with recovery of pruned functionality.

Critical strengths
  1. The recovery-aware selection is useful. The method considers weight reconstruction when choosing what to prune. The small-model ablation supports this design. The complete method gains about 7.2 and 6.6 percentage points over repaired uniform allocations at two budgets. This suggests that selection adds value beyond reconstruction alone.
  2. The compressed models offer practical benefits. The method produces dense networks that can run on conventional hardware. Reported gains include roughly 1.3–2× GPU throughput and 1.9–3.2× CPU speedups. Tests cover several backbones and twelve tasks. This provides broader support than parameter savings alone.
Critical weaknesses

The evaluation acknowledges different checkpoint-selection protocols for SliceGPT/OSSCAR. The differences are not explained sufficiently. Test data are excluded from architecture selection. However, the split used to select classifier checkpoints is less clear. Some accuracy margins are small. A common selection rule could change the ranking. This is an ambiguity requiring clarification.

Questions for authors

Which split selects classifier checkpoints for each method? What differs for SliceGPT/OSSCAR? Would a common checkpoint-selection rule change the main accuracy comparisons?

3 comments

We thank the reviewer for his feedback. We are glad the reviewer finds Carve VFM a promising practical compression framework with substantial empirical coverage, and recognises its main contribution. We address the concern below.

Q: Which split selects classifier checkpoints? What differs for SliceGPT/OSSCAR? Would a common rule change the comparisons?

We have removed this ambiguity.

Shared protocol. For every method, the compressed backbone is frozen and a fresh linear head is trained with a fixed 10-epoch AdamW schedule. 20% of the training split is held out as validation, and the checkpoint is chosen on validation among epochs {1, 3, 5, 10}. The test split is never used for architecture search, calibration, repair or checkpoint selection.

What differed. The only difference is how the 20% validation subset is drawn. In the original script it was re-sampled at random every epoch. SliceGPT-ViT and OSSCAR-ViT were run later with our revised script, which draws one stratified 20% subset and keeps it fixed for the whole run. Search, budgets, calibration sets, seeds and head schedule are identical.

This does not affect the ranking. To remove checkpoint selection entirely, we recomputed every method at the fixed final epoch (epoch 10) from the existing logs (about 2,950 runs). Epoch 10 was already the chosen checkpoint in 81–94% of runs for every method, and the 12-task means change by 0.00–0.23 pp (about 0.1 pp on average). The change is no larger for Carve VFM than for the baselines.

SettingCarve VFM (chosen → epoch 10)Strongest other (chosen → epoch 10)
ViT-B 20M54.3 → 54.2SVD-LLM 51.0 → 51.0
ViT-B 60M90.8 → 90.8SVD-LLM 89.7 → 89.7
ViT-S 14M85.8 → 85.7CORP 83.1 → 83.1
ViT-S 6M49.5 → 49.4SliceGPT-ViT 40.5 → 40.5
MetaCLIP-2 60M86.6 → 86.4CORP 83.6 → 83.5
ViT-L 100M83.8→ 83.8SVD-LLM 84.9 → 84.9
ViT-L 200M91.7→91.7SVD-LLM 91.7 → 91.6
MetaCLIP-2 20M48.6 → 48.5SliceGPT-ViT 53.4→ 53.4

The ranking is unchanged in all eight settings, including the two where Carve VFM is not first.

Direct check. We also re-ran Carve VFM with the revised fixed-validation script (ViT-S 14M, seed 17). CIFAR-100 scored 79.7 on validation and 80.2 on test (chosen: 80.5). CUB-200 scored 77.8 on validation and 78.6 on test (chosen: 79.1). The differences of 0.3–0.5 % are far smaller than the gaps between methods in this setting.

Revision. The final version will state the validation protocol explicitly and report all methods using it.

I thank the authors for the detailed reply. My major concerns have been addressed. I will change my score to accept.

We thank the reviewer for the constructive feedback and for updating the score. The revised version will describe the evaluation protocol clearly.

Reviewer_YPy6OpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper study VFM specialization, where given a pretrained vision model, the downstream task and a small sample of labeled calibrations dataset and a paramater budget, without retraining the backbone model, the method extract a small dense model from the pretrained model that is specialized to the task. They proposed a method Carve VFM that contain three stages, including passing the calibration set through the frozen model to get task-specific stats, evolutionary search over the per-layer parameter retention ratio in different modules guided by well-known OBS method, and carving the selected architecture and repairing each layer with a closed-form regression. Their experiment result indicate that the condense model people need is inside the model but only need to be carved out.

Critical strengths
  1. The problem setting is practical and, from the efficiency perspective, clearly define. Given a VFM, a task, and a parameter budget, the method returns an ordinary dense model, with no masks, low-rank branches, or sparse kernels left at inference. This make the output easy to deploy, and the deployment study in the paper supports this as author indicate.
  2. The paper is transparent about their limitations which is great and valuable. They reports the higher batch 1 GPU latancy compare with the full model, the cases where native compact models win and the high variance at tight budgets and how often the audit falls back to the global fitness.
Critical weaknesses

W1. The paper does not compare carved models with the smaller models that DINOv3 already releases.

The author presents carving as an alternative to distillation, but the problem is that DINOv3 already releases distilled models at several sizes, so as to some other model families. DINOv3 releases ViT-S (21M), ViT-S+ (29M), ViT-B (86M), and ViT-L (300M), and also releases ConvNeXt-T (29M), ConvNeXt-S (50M), ConvNeXt-B (89M), and ConvNeXt-L (198M).

And thanks to their providing of different model size, because if I had a 35M budget on model size, the first thing I would do is to download ViT-S+ or ConvNeXt-T from the DINOv3 release rather than compress a larger model to fit the budget. So for me the bar for Carve VFM is simple, that the carved model has to beat what I can already download at the same size.

The paper already reflects part of this comparison in Table 1, but to make it more concrete, I put the numbers side by side in the table below, where each model your method carves from a larger DINOv3 model is compared against both the unpruned smaller DINOv3 model and your method applied to that smaller model, all within the same budget:

Source →\to budgetCarve VFMUnpruned smaller DINOv3 modelCarve VFM on a smaller DINOv3 sourceΔ (Carve - best smaller-source option)
ViT-B →\to 60M90.8ViT-S (21.6M): 87.2ViT-S →\to 14M: 85.8+3.6
ViT-B →\to 35M80.4ViT-S (21.6M): 87.2ViT-S →\to 14M: 85.8-6.8
ViT-B →\to 20M54.3None within the budgetViT-S →\to 14M: 85.8-31.5
ViT-L →\to 200M91.7ViT-B (85.7M): 90.6ViT-B →\to 60M: 90.8+0.9
ViT-L →\to 150M90.7ViT-B (85.7M): 90.6ViT-B →\to 60M: 90.8-0.1
ViT-L →\to 100M83.7ViT-B (85.7M): 90.6ViT-B →\to 60M: 90.8-7.1

We can see that carving the larger model only clearly pays off at ViT-B →\to 60M. At the tight budgets your paper emphasizes, simply starting from a smaller DINOv3 model is roughly 7 to 31 points better. And this is before counting the released models that also fit these budgets but are not evaluated: ViT-S+ and ConvNeXt-T (29M) from 35M upward, ConvNeXt-S (50M) from 60M upward, ConvNeXt-B (89M) from 100M upward, and ConvNeXt-L (198M) at 200M. This comparison is standard in structured-pruning work (e.g., CoFi [1], ZipLM [2], NViT [3], and Isomorphic Pruning [4] all compare against native or distilled models of similar size).

Because this might seriously affect the validity of the value of this paper, I kindly ask the authors:

Q1. How do the carved models compare with the other released DINOv3 models (ViT-S+ and the four ConvNeXts) under the same linear-probing protocol as Table 1?

Q2. Where do the carved models sit on the accuracy-vs-parameters and accuracy-vs-FLOPs curves relative to the released DINOv3 models?

Q3. In Table 10, how much of the gain comes from carving, and how much comes from DINOv3 pretraining itself? Table 10 only compares against models with different pretraining data.

W2. The three claimed contributions are close to existing work, and most of the closest work is not cited.

The paper presents task-specific allocation, the recoverability-aware criterion, and the closed-form repair as its three main contributions in the method. However, I found fairly direct precedents for each of them, and most of these are missing from the paper:

(1) Task-specific allocation under a budget: Searching per-layer compression levels under a budget is well studied: SPDY [5] uses dynamic programming over layer-wise error scores, and EvoPress [6] runs an evolutionary search over a database of per-layer compression levels, which is very close to your Stage 2. For ViTs, NViT [3] already redistributes heads, QK/V dimensions, and MLP width across blocks, including per-head QK and V pruning similar to your operator C.

(2) Recoverability-aware criterion: Eq. (2) is identical to the structured OBS [7] saliency in Eq. (2) of ZipLM [2], and your removal procedure (remove one group, refit the surviving weights, update G^{-1} by a Schur complement) is ZipLM's Algorithm 1, applied to the same groups: attention heads and MLP down-projection columns. Since an OBS cost already assumes an optimal refit of the remaining weights, it is recoverability-aware by construction, and SlimGPT [8] uses the same idea for LLMs. Note that the paper attributes this procedure to OBC, but the structured version comes from ZipLM.

(3) Closed-form repair: Eq. (6) is essentially what your own baseline CORP [11] does: it predicts the removed components from the retained ones with a closed-form ridge regression and folds the compensation into the weights. Least-squares refitting after channel removal goes back to He et al. [9] and ThiNet [10], and QERA [12] gives a closed-form low-rank version for quantization.

Beyond these three points, two uncited works study nearly the same setting: TACO [13] specializes ViTs and ConvNeXts from a few task samples (but with a uniform ratio per layer), and Kwon et al. [14] follows the same search-then-least-squares recipe on BERT in under 3 minutes. EvoPress [6] also shows that summed per-layer errors stop predicting the true error as compression grows, which matches your audit switching to the global fitness in up to 89% of runs. So the new parts seem to be the non-uniform allocation search with the per-layer H/C choice and the adaptive audit, and the gains (5 wins, 1 tie, and 2 losses in Table 1 when SVD-LLM is counted) are not strong enough to carry the paper on their own.

Q4. Could you position the method against TACO [13], Kwon et al. [14], ZipLM [2], and CORP [11], and state which parts are new?

Q5. Could you ablate the new parts: local-only vs. global-only vs. adaptive fitness, and H-only vs. C-only vs. mixed operators?

Q6. Why truncate the correction in Eq. (6) to rank r, given that it is merged into a dense matrix of the same shape?

Small Problem: The information density of Figure 2 and 3 (especially 2) is a bit too much and overwhelmed. The goal of the figure is to help understand rather than complicateing it, so I suggest the author to remove a bit some of the redundant indication in the next version that is already well-defined in the text.

  • [1] Structured Pruning Learns Compact and Accurate Models
  • [2] ZipLM: Inference-Aware Structured Pruning of Language Models
  • [3] Global Vision Transformer Pruning with Hessian-Aware Saliency
  • [4] Isomorphic Pruning for Vision Models
  • [5] SPDY: Accurate Pruning with Speedup Guarantees
  • [6] EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
  • [7] Second Order Derivatives for Network Pruning: Optimal Brain Surgeon
  • [8] SlimGPT: Layer-wise Structured Pruning for Large Language Models
  • [9] Channel Pruning for Accelerating Very Deep Neural Networks
  • [10] ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
  • [11] CORP: Closed-Form One-Shot Representation-Preserving Structured Pruning for Transformers
  • [12] QERA: An Analytical Framework for Quantization Error Reconstruction
  • [13] TACO: Vision Models Can Be Efficiently Specialized via Few-Shot Task-Aware Compression
  • [14] A Fast Post-Training Pruning Framework for Transformers
Questions for authors

Please answer the question mentioned in the Weaknesses section. Notice that there are a tons of concurrent work in the pruning domain and it is not possible to request the author to make the comparison to every similar method, but still I would like to see if author can still position the work and validate the value, given the inclusion of so many related works, and I am willing to consider adjusting my score after the rebuttal.

5 comments

Response, Part 1 of 2: W1 and related questions (Q1–Q3)

We thank the reviewer for their thorough review and for suggestions that helped us strengthen and clarify our claims. Below we answer each concern and report the new experiments we ran.

CARVE is proposed as an efficient alternative to distillation. CARVE produces a specialized model in minutes on a single GPU from only 512 task samples, with no gradient training. Distillation trains the student on the full pretraining corpus: the released DINOv3 models were each distilled from a 7B teacher on ~1.7B images. Comparing a model built from 512 samples in minutes with models built from billions of samples is therefore not a like-for-like comparison.

The cited works (CoFi, ZipLM, NViT, Isomorphic Pruning) compare against distilled models because they operate in the same regime as distillation. NViT and Isomorphic Pruning prune iteratively and then fine-tune for ~300 ImageNet epochs with distillation. CoFi learns its masks during full task fine-tuning with distillation. Their data, compute and time are comparable to distillation itself.

To separate the effect of data from the effect of the method, we answer in three parts:

  • (i) How does CARVE compare with the cited methods in the limited-data regime?
  • (ii) Is the remaining gap to the distilled models due to carving itself, or to the training budget and data?
  • (iii) In which practical settings is CARVE better than distillation?

(i) CARVE vs. the cited methods in the limited-data regime. CARVE targets data and compute efficiency, so we ran the cited methods in this regime. Each used its own pruning criterion and received 100 distillation steps on the same 512 images.

Table R1. Accuracy (%), ViT-B source, mean over 12 tasks × 3 seeds, same linear probe. CARVE uses 0 gradient steps; all other methods use 100.

BudgetIsomorphicNViTCoFiTACOCARVE
20M34.228.537.738.552.8
35M64.479.458.173.880.0
60M89.589.984.187.390.1

With no gradient updates, CARVE is the best method at every budget, and it leads every method by more than 14 points at 20M. NViT, which prunes iteratively with training, is close at 35M and 60M. Results for the other sources in our main table are being run and will be added to the paper.

(ii) Q1/Q2: CARVE vs. distilled models, with and without fine-tuning. The released models are indeed better at some sizes, and the question is why. To answer it, we gave the carved models a small training budget to test the following: If carving had permanently damaged the model, a brief fine-tuning could not close the gap. If the gap comes from the difference in training budget, it should.

Table R2. Accuracy (%), mean over 12 tasks × 3 seeds, grouped by size.

SizeReleased (distilled on LVD-1689M)CARVE (512 images)CARVE + task fine-tuning
~20MViT-S 88.0ViT-S+ → 20M 87.889.4
~30MViT-S+ 88.9, ConvNeXt-T 84.7ViT-B → 35M 80.086.4
50–60MConvNeXt-S 85.0ViT-B → 50M 88.7; → 60M 90.189.2; 90.4 [(ViT-B: 90.0)]
150–200MConvNeXt-L 84.5ViT-L → 150M 88.8; → 200M 90.8-
  • Without any training, CARVE beats the released ConvNeXt models at 50M and above.
  • With fine-tuning, the carved models close most of the gap to the released ViTs. At 35M, the gap to ViT-S+ shrinks from 8.9 to 2.5 points, without the model ever seeing the 1.7B-image corpus. The carved model is therefore a strong starting point: most of the gap reflects the difference in training, not a weakness of carving.
  • We will add accuracy-vs-parameters and accuracy-vs-FLOPs curves in the paper; both give the same picture.

(Rest is given in the next comment)

(iii) Practical settings where CARVE is better than distillation. The comparison above assumes a distilled family exists. Often it does not, and this is where CARVE matters most.

  • Domain foundation models. These are usually released in a single size. Under a tight budget, the only alternative is a general-purpose small model, which loses the domain pretraining. Carving RAD-DINO (86.6M) to 35M keeps it. The carved model beats the released ViT-S+ (28.7M) by 7–10 points:

    • PneumoniaMNIST: 95.8 vs 87.2
    • RSNA: 75.2 vs 67.3
    • CheXpert-5: 65.4 vs 55.9
  • Systems built on the original model. A distilled model has its own embedding space, so every head, adapter, retrieval index and stored embedding built on the original must be rebuilt. CARVE repairs the pruned model to reproduce the original features, so these components keep working. With the original ViT-B linear head applied unchanged, CARVE ViT-B → 60M reaches 86.1% (original: 87.4%), while the released ViT-S reaches 4.1%.

  • Large models. Distilling an intermediate-size model at billion-parameter scale is prohibitively expensive, so model families leave large gaps between sizes. Qwen2.5-VL has no release between 3.75B and 7.75B. CARVE fills this gap from the 7B release with a few hundred samples:

Table R3. Accuracy (%), 3 seeds. Bold: carved model beats the native 3B.

TaskNative 3B (3.75B)CARVE 7B → 4BCARVE 7B → 5.5BNative 7B (7.75B)
Aircraft90.783.192.493.8
CUB-20095.294.396.296.3
Pets97.598.098.098.6
DTD91.192.692.893.8
CIFAR-10091.192.493.994.8
MIT Indoor98.098.398.498.6
ScienceQA87.285.989.993.4
Mean (7 tasks)93.092.194.595.6
  • CARVE 7B → 5.5B beats the native 3B on every task, and closes more than half of the gap to the native 7B.
  • CARVE 7B → 4B wins 4 of 7 tasks against the native 3B, with a slightly lower mean (92.1 vs 93.0) that comes mostly from Aircraft.

Q3. Our main table also includes MetaCLIP, which uses a different pretraining scheme: a contrastive image–text loss, while DINOv3 is self-supervised. and Carve is best in 2/3 settings in MetaCLIP as well.

In summary, CARVE is not meant to replace large-scale distillation, but to provide a cheap alternative when distillation is too costly or unavailable. We will add these results, the curves and the carving guideline to the paper. Related work now covers every work listed, and Table 1 compares the closest ones.

Response to W2 (Q4–Q6):

Response to W2 (Q4–Q6)

We thank the reviewer for the detailed assessment and references. Eq. (2) is now attributed to ZipLM, and the revised Sec. 2, Table 1 and Appendix H discuss every work listed.

As the reviewer points out, the pruning literature is rich, and CARVE uses ideas from several of these papers, most directly the OBS solver of ZipLM. We do not contribute another solver. Our objective is to trade the generality of a VFM for fast compression. CARVE starts from a general model and keeps only what one task needs, to obtain a smaller model for that task. It must do so cheaply: a few hundred labeled images and minutes on one GPU, with no training of the backbone.

Most cited works have a different objective: they compress a model while keeping what it already does. ZipLM and Kwon et al. start from a model fine-tuned on the task. CORP, NViT and EvoPress keep a general model general. TACO is the one cited work with the same objective as ours, and we compare against it directly (Sec. 4 below). We also compare against the other methods, while recognizing that they were designed for a different objective than ours. Each design choice serves this objective as shown below.

1. Choose the architecture by how well it can be repaired. Every pruned model is refit in closed form, so what matters is accuracy after the refit. OBS applies this criterion to choose channels or heads within a layer. CARVE extends it to the two decisions OBS leaves open: how to split the parameter budget across layers, and whether a layer loses whole heads or has all its heads narrowed. These decisions matter, since the same refit on three architectures gives very different accuracy (ViT-S at 14M / 6M, Sec. 4):

Architecture chosen byAfter the same repair
Importance (FLAP)63.5 / 36.7
Uniform ratio78.2 / 47.7
CARVE search85.4 / 54.3

An importance-selected subnetwork ends up below a uniform one. This is the difference to CORP: both use a closed-form ridge solve, but CORP applies it to an architecture fixed beforehand by simple ranking criteria, while CARVE selects the architecture for its repairability. CARVE beats CORP in all eight settings of Table 2, by 2.7 to 23.8 points.

2. A fitness that is efficient in labels and in compute. We examined which score can rank candidates in our setting.

  • Held-out probe accuracy is the objective we care about. With few labels it is noisy (up to 3% std), and each evaluation requires carving, repairing and probing a model.
  • Summed OBS errors are computed from activations and cost a table lookup. They ignore cross-layer effects and the task.

No single score is adequate alone, so CARVE combines the two. The summed OBS error ranks the population. Held-out accuracy tests this ranking on 12 candidates, and replaces it only when the two disagree. The switch happens in 26% of mild and 51% of tight ViT-S/B runs, 79% of MetaCLIP-2 runs and 89% of ViT-L runs. EvoPress reported for LLMs that summed layer errors can mislead; and our results here confirms it.

3. A per-layer attention operator. Attention can be reduced by dropping whole heads (H) or by narrowing all heads (C). Which one the refit recovers better depends on the layer, so CARVE chooses per layer, jointly with the allocation: The choice matters as shown in analysis Fig.6 and supported by ablation (ViT-S, 14M / 6M): H only 84.6 / 51.8, C only 85.4 / 54.2, per-layer 87.3 / 57.4. ZipLM and Kwon et al. use H only; NViT mixes both, but decides during training.

4. Positioning and what is new. CARVE differs in three ways: (i) heads, Q/K, V/O and MLP widths are reduced jointly without backbone training; (ii) the architecture is searched and scored after repair, not before it; (iii) the cheap OBS ranking is audited on held-out task accuracy. In the same data-efficient regime (ViT-B, 12 tasks, 3 seeds), CARVE without any distillation steps matches or exceeds NViT and TACO with 100 distillation steps at 60M and 35M parameters, and leads by at least 14 points at 20M (Table in W1 reply).

Q5. Ablations (ViT-S → 14M/6M). Scoring every candidate on held-out accuracy is the most accurate but builds all 120–200 candidates, against 32 for the audit. At 14M all fitness variants are within 0.8 pt; at 6M the local score alone loses 5.8 pt and the adaptive audit recovers 4.3.

Q6. Rank. The rank came from EoRA, where the correction is a separate low-rank path and the rank limits its size; once merged, that benefit is gone. We kept it because we assumed a full matrix estimated from few images would overfit. It does not, because the regression uses ~100k tokens: full-rank repair improves over r=128 by up to 8.9 pt at tight budgets. Full rank is the new default, simplifying repair to plain least-squares.

Gains. CARVE leads in 7/8 dense settings (Table 2) and 9/12 settings vs. SVD-LLM while being 1.4–2.3× faster.

Figs 2–3. are updated for clarity.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reading, constructive feedback, and engagement during the discussion. Your comments directly contributed to making this paper stronger and clearer.

We have uploaded the revised version, which includes the updates discussed in our responses:

  • Full-rank repair. We especially thank Reviewer YPy6 for pointing to the rank of the correction. Following this suggestion, we made full-rank repair the default. This simplifies the method to a plain least-squares solve and improved our results, most notably at tight budgets.

  • Comparison with released and distilled models. We added the comparison with the released DINOv3 models, the limited-data baselines, and the settings where no distilled model exists (a single-size domain model and a large multimodal model).

  • Evaluation protocol. All methods are now reported at the fixed final epoch, and the validation protocol is stated explicitly.

  • Positioning and ablations. Related work now covers the suggested references, and we added the ablations on the fitness and the attention operator, together with the definitions of the decoupling variants.

  • Presentation. Figures 2 and 3 were simplified for clarity.

We are grateful for your time and for the suggestions that improved this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.