- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper aims to get a small, task-specific dense model out of a big VFM, given a labeled calibration set (512 images) and a parameter budget . Carve VFM works in three steps:
- It first caches OBC-style greedy removal curves for every layer and operator: MLP channels, whole heads (H), or within-head coordinates (C).
- It then runs an evolutionary search over per-layer MLP/attention retention and the operator . The search is scored by a cheap additive "recoverable damage" proxy, and falls back to held-out linear-probe accuracy when an audit says the proxy's ranking can't be trusted.
- Finally, it carves the chosen architecture and folds a ridge + SVD-truncated correction into the surviving weights, so nothing extra remains at inference.
Evaluation covers 4 backbones (DINOv3 S/B/L, MetaCLIP-2 B), 12 classification tasks and 3 budgets each, mostly with frozen-backbone linear probing. There is also a full fine-tuning comparison against compact ImageNet models and CPU/GPU throughput measurements on ViT-L.
Critical strengths
1. What comes out is a plain dense model, and the speedups are real. No masks, no sparse kernels, no extra low-rank branches. That's why Table 11 can show actual wall-clock gains (– GPU, – CPU throughput vs. full ViT-L) instead of theoretical FLOP savings. I also admitted that the authors didn't hide the awkward part: batch-1 GPU latency actually goes up slightly.
2. The experiments are broad and reported in a way I could actually dig into. 4 backbones 12 tasks 3 budgets, three seeds with stds, per-dataset tables, wall-clock costs, and a limitations section that is fairly upfront.
Critical weaknesses
1. It is unclear whether carving a larger VFM is preferable to using a smaller pretrained model from the same family. Table 1 shows that DINOv3 ViT-B carved to 20M achieves 54.3% mean accuracy, while the original ViT-S (21.6M) achieves 87.2%. Similarly, carving ViT-L to 100M gives 83.7%, below the original ViT-B (85.7M) at 90.6%. Although the budgets are not exactly matched in the first comparison, these results raise an important practical question: why carve a larger model when a smaller pretrained model may already perform substantially better? I would like to see a more direct comparison within the same pretrained model family.
2. It is hard to tell which component actually drives the improvements. In the decoupling experiment, adding repair to uniform pruning increases mean accuracy from 31.7% to 78.2%, while the full method reaches 85.4%. This suggests that recovery accounts for much of the improvement. Meanwhile, the adaptive search switches to global guidance in 79% of MetaCLIP-2 runs and 89% of ViT-L runs. How much does the recoverability-aware local fitness really contribute on these larger models?
Questions for authors
1. How are checkpoints selected? Probes and fine-tuning are evaluated at epochs 1/3/5/10, and Table 10 reports the "best checkpoint." Best according to which split? The search stage explicitly avoids the test set, but I couldn't find the same statement for downstream checkpoint selection. If it were test accuracy, that would be a real problem for the whole evaluation, so a clear statement (and ideally fixed-epoch results, e.g., epoch 10) would settle this. This one could change my score.
2. Is CORP's DTD number expected? At the mild budgets, CORP gets 57.9–63.8% on DTD across all four backbones, while Carve and SVD-LLM are around 75–83% on the same settings. On ViT-S at 14M, DTD alone accounts for roughly half of Carve's 2.7-point margin over CORP. Could you double-check this configuration? More generally, were the baselines run with their original recovery/compensation steps? Several (e.g., SNIP–Mag., SnapViT) end up at or below Random, which surprised me.
3. What exactly is removed in the "without repair" setting? The decoupling paragraph reports 28.2% for FLAP subnetworks without repair at 14M. But Table 1's FLAP on the same three datasets (C100, CUB, BloodMNIST) averages 57.2%. Also at 14M, uniform without repair (31.7%) is below Random on those datasets (40.6%, from Table 3), although at 6M it is above (31.5% vs. 27.6%). Is FLAP's own bias compensation stripped here? Does "uniform without repair" still use the OBC removal order? I need this to judge the repair-vs-search split in Weakness (2).
3 comments
The authors have satisfactorily addressed my concerns in their rebuttal. I appreciate their clarifications and additional explanations, which have resolved my remaining questions. I am satisfied with the response and will update my recommendation to Accept.
We thank the reviewer for the discussion and for raising the score. The revised version includes these clarifications.
We thank the reviewer for the careful reading, and for noting the deployment value and the transparency of the paper.
Q1. Checkpoint selection. Details are in our reply to Reviewer BxTr. For every method, the checkpoint is chosen among epochs {1, 3, 5, 10} on a validation subset of 20% of the training split; the test split is never used for selection. The paper's wording ("full training set") omitted this and will be corrected. The only difference is how the subset is drawn: the original script re-sampled it every epoch, while SliceGPT-ViT and OSSCAR-ViT, run later, used a revised script that keeps one stratified subset fixed. To remove this difference, all methods are now reported at the fixed final epoch (epoch 10), as the reviewer suggested. Epoch 10 was already the chosen checkpoint in 81–94% of runs, 12-task means change by 0.00–0.23 pp, and all eight rankings are unchanged (table in our reply to Reviewer BxTr). For full fine-tuning (Table 11) at epoch 10, the 14M specialist reaches 87.5% vs. 83.4% for the strongest native compact model (+4.1 pp).
W1. Carving vs. smaller pretrained models. See our reply to Reviewer YPy6 (Tables R1–R3). When a distilled model of the target size exists, it is preferable at aggressive budgets. Those models were distilled from a 7B teacher on ~1.7B images, while Carve VFM uses 512 task images and minutes on one GPU; a short task fine-tuning closes most of the gap. It is most useful where no suitable distilled model exists: single-size domain models (RAD-DINO → 35M beats ViT-S+ by 7–10 pp), systems built on the teacher (the teacher's linear head transfers unchanged at 81.8% vs. 4.1% for native ViT-S), and large-model gaps (Qwen2.5-VL 7B → 5.5B beats the native 3B on all 7 tasks).
W2. Which component drives the gains? The search is repair-aware: every candidate is scored by its accuracy after repair, so architectures are selected for how well repair can recover them. Repair therefore looks like the main driver, but much of its benefit comes from an architecture chosen for it. Holding repair fixed isolates the search: a FLAP-selected architecture reaches 63.5%, a uniform one 78.2%, and the repair-aware search 84.9% (ViT-S, 14M; CIFAR-100, CUB-200, BloodMNIST; seed 17). That is +6.7 pp from search alone. (The paper's +7.2 pp used the three-seed Carve mean; we now use the matching seed.)
On the fitness: the local OBS score ranks the population cheaply (32 candidate builds instead of 120–200), and held-out accuracy audits the top 12, overriding the ranking when they disagree. The high switch rate on ViT-L (89%) and MetaCLIP-2 (79%) shows why the audit is needed on larger models. On ViT-S, local-only loses 5.8 pp at 6M and the adaptive audit recovers 4.3 pp.
Q2. CORP on DTD. The CORP rows are our re-implementation of the CORP recipe (Wanda-scored MLP channel pruning and QK-dimension pruning with closed-form logit compensation and bias correction), because no official code was available at submission time. The authors added a repository link in their latest version (9 Nov 2026, after our submission). Its DTD accuracy is low on all four backbones at the mild budget (57.9–63.8%), so this is systematic and not a single-run artifact; we will run it with the new available official code and see whether it is specific to our re-implementation.
Recovery steps. FLAP and CORP keep their own compensation. Random, SNIP–Mag. and SnapViT use the same uniform per-layer MLP sparsity and bias correction, and differ only in which attention heads are removed under a global head budget.
Baselines at or below Random. We attribute the gap to how head removal is distributed. SNIP–Mag. and SnapViT leave at least one layer with a single head (of six) in every run (24/24 at ViT-S 14M), concentrating removal in a few layers. Random keeps at least 3 heads in every layer in half of the runs.
Q3. What "without repair" removes. We will add these definitions to the appendix. All variants share calibration data, budget projection and parameter budget.