acceptodds
Under review as a conference paper at ICLR 2027

Channel-Remix: Data efficient Synthesis of Task-Specific Compact Models by Remixing Vision Foundation Model Channels

Abstract

Large-scale pretraining endows vision foundation models (VFMs) with rich, transferable visual knowledge, yet their compute and memory costs put much of it out of reach in practice. Knowledge distillation compresses this knowledge by transferring it to a compact student under the VFM's supervision. However, while such supervision provides a strong training signal, it leaves the student's parameter space unconstrained. Consequently, bridging the gap between teacher representations and viable student weights typically demands substantial task data and long training. We propose Channel-Remix, which uses the VFM not only as a teacher but as a library of pretrained weight directions from which the student is built. This recasts knowledge transfer as selecting and recombining channels that pretraining has already learned, a problem that requires far less task data than learning new weights. Given a parameter budget and a small calibration set, Channel-Remix retains a core subset of channels and sparsely mixes pruned ones into them to form new task-adapted channels with zero runtime cost. By grounding both the learning signal and the student's parameter space in the VFM, Channel-Remix achieves high data efficiency, matching LoRA-based distillation using 3-4x less calibration data. Across 16 tasks, 7 pretraining families, and 5 model sizes, it improves pruned models by 4.6% on average and closes the gap to the original VFM to under 2% at 50% compression, outperforming distillation baselines under full fine-tuning, PEFT, and post-pruning reconstruction while training fewer parameters. Moreover, because every student is built from the same pool of VFM directions, models for different tasks become directly comparable, opening a route to analyzing which knowledge is shared across tasks and which is task-specific.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Vision foundation models, model compression, data-efficient adaptation, parameter-efficient learning, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Jxn2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Channel-Remix uses a vision foundation model as both a teacher and a source of pretrained channels. After structured pruning, it uses a small calibration set to sparsely mix discarded donor channels into retained anchor channels, and then folds these mixtures into standard weights without adding inference cost.

Critical strengths

The authors claim that constraining adaptation to a sparse span of frozen VFM channels and folding them into a pruned backbone gives a cleaner inductive bias for scarce-data transfer than reducing parameter count alone, with no runtime overhead. They also state that the evaluation is broad and that, under matched low-data schedules, sparse remixing (K=3/5/7) is stable where dense mixing collapses and generally outperforms LoRA/PiSSA.

Critical weaknesses

Already stated in the Questions.

Questions for authors
  1. In lines 92-93, the authors describe “A small calibration set.” Could the authors clarify whether this small calibration set is in fact more important than channel-remix? In addition, please explain the rationale for selecting the specific calibration sets in Table B.1.
  2. The authors state in the appendix that these calibration sets are class-balanced. However, it is unclear whether the distribution of this calibration set is actually balanced. If it is, how was this balance determined or verified? If the distribution is inherently skewed, how can the generalization ability of the resulting small, compact model after transfer be guaranteed? In addition, would additional task data be needed to continue SFT on this compact model? I would appreciate clarification on these points.
  3. How large is the impact of LKD\mathcal{L}_{\mathrm{KD}}? Is it possible that the good performance in this experiment stems more from KD? In Table 2, does “teacher guidance” refer to the KD loss term? Clarification on these points would be appreciated.
  4. Is there a table comparing the training cost with that of other methods?
3 comments

We thank the reviewer and appreciate that he recognizes the main intuition and motivation of the paper and also values the breadth of the evaluation. We answer the reviewer questions below:

Q1. Is the calibration set more important than Channel-Remix? The reviewer raises a very good point. With so few images, it is natural to wonder how much of the gain comes from the data versus the method. Every method in our comparison uses the same calibration images, compressed model, teacher, and schedule, so accuracy differences reflect the adaptation method, not the data. Two results confirm this:

  • Channel-Remix beats LoRA-8 in 85/96 matched pairs (p<1e-12) with the same 512 images, and matches LoRA trained on 3–4× more data (Fig. 4).

  • Across three calibration draws (seeds 17, 42, 91) on ViT-B/35M, 12 tasks: Remix 87.0 / 86.7 / 87.0%, LoRA-4 85.0 / 85.6 / 85.6%. both are stable and the gap holds each time.

To clarify, Table B.1 lists the full dataset statistics; the calibration sets are class-balanced subsets drawn from the training split, not separate curated sets. The 16 datasets were chosen for domain diversity so we can confirm the method is not restricted to one domain and analyze how donor selection changes across domains, as in the cross-task analysis (Sec. 5).

Q2. Class balance, generalization, and further fine-tuning. Studying the effect of calibration set composition is an interesting direction. To answer the reviewer's specific questions:

  • Is it actually balanced? Yes, the sampler gives each class the same number of images (e.g., CIFAR-10 at N=512: 51–52 per class; DTD: 10–11 across 47 classes). We record the per-class counts in every run and now include them in the appendix.

  • Why equal per class? Simply to make sure every class has at least a few images. Random draws could miss some classes entirely, which would hurt the performance on those. Also, in our setting, we assume only a small set of images is available. Following the training set distribution would require knowing it in advance, which could be seen as using information from the full dataset. Equal sampling avoids this and ensures every class is represented.

Generalization? We did not test skewed sampling because it requires knowing the training distribution, which in our setting we assume we do not have. What we can confirm is that the method is stable across different balanced calibration sets (seeds 17/42/91: Remix 87.0/86.7/87.0%, LoRA-4 85.0/85.6/85.6% on ViT-B/35M, 12 tasks), and generalization is measured on the test sets of all 16 datasets, which are never used during adaptation.

  • Further fine-tuning? The folded model is a standard network with no extra modules, the performance is as reported and already competitive with methods that use 3–4× more data (Fig. 4). No further fine-tuning is needed, but since the model is standard, it can be fine-tuned further if wanted and more data becomes available.

Q3. Role of KD: Yes, "teacher guidance" in Table 2 refers to the KD and feature-matching losses. The reviewer is indeed right that KD helps, with only a few hundred images, per-layer output and representation matching prevents the compact model from overfitting. But we made sure its effect is controlled for a fair comparison:

  • Every iterative baseline (LoRA, DoRA, PiSSA, SSF, full fine-tuning, Dense Remix) uses the same KD and feature-matching losses, same weights, same temperature.
  • Removing the donors while keeping KD loses 17.3% of the 28.5% recovered (Table 2a), the gains come from the donor channels, not from distillation solely.

Q4. Training cost comparison. We reported the training cost in App C.5. We did not highlight it because all methods have similar cost, it is mainly dominated by forward passes through the model. At 40 epochs on ViT-B/35M with 512 images (single GPU): LoRA-4 ~3.6 min, LoRA-8 ~3.8 min, full fine-tuning ~4.6 min, PiSSA-8 ~4.0 min, Channel-Remix ~3.8 min. Where cost really differs is data: to match Remix at 512 images, LoRA needs more data (Fig. 4), which scales training time proportionally (LoRA at 2048 images take ~15 min). Channel-Remix achieves the same accuracy in ~4 min.

Most of my questions have been resolved, and I will maintain my rating.

We thank the reviewer for the follow-up, and for the time and effort put into reviewing our paper. We are glad that our responses resolved most of the questions.

If any point remains open, or if there is any additional analysis or clarification that would make the paper a clear accept in the reviewer’s view, we would be happy to provide it.

Reviewer_nivDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Channel-Remix compresses vision foundation models by retaining anchor channels and learning sparse combinations of discarded donor channels. The combinations fold into standard weights, improving compressed-model accuracy across 16 recognition tasks without additional inference overhead.

Critical strengths
  1. Sparse recombination provides a practical way to recover discarded pretrained knowledge while preserving the compressed architecture.
  2. Evaluation across seven pretraining families and multiple calibration budgets supports the method’s breadth.
  3. Donor-selection ablations and cross-task sharing analyses help explain the learned adaptation.
Critical weaknesses
  1. The final linear probes use full task training data, so the reported calibration efficiency does not establish end-to-end label efficiency.
  2. Severe failures of Dense Remix, Ridge, and Neuron Merging warrant stronger diagnostics before attributing their performance primarily to insufficient data.
  3. Final active-coefficient counts exclude temporary search allocations and therefore understate the resources needed during donor discovery.
  4. Evaluation remains limited to recognition tasks, leaving applicability to detection and segmentation unestablished.
  5. The related-work survey insufficiently covers vision-specific adaptation and efficient tuning [1-10].

[1] Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation. CVPR. 2025.

[2] AiRs: Adapter in Remote Sensing for Parameter-Efficient Transfer Learning. IEEE TGRS. 2024.

[3] Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach. CVPR. 2024.

[4] 5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks. CVPR. 2025.

[5] V-PETL Bench: A Unified Visual Parameter-Efficient Transfer Learning Benchmark. NeurIPS. 2024.

[6] Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model. ICLR. 2024.

[7] Parameter-efficient is not Sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions. ACM Multimedia. 2024.

[8] Efficient Adaptation of Pre-trained Vision Transformer underpinned by Approximately Orthogonal Fine-Tuning Strategy. ICCV. 2025.

[9] TEA: A Training-Efficient Adapting Framework for Tuning Foundation Models in Remote Sensing. IEEE TGRS. 2024.

[10] Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation. CVPR. 2024.

Questions for authors
1 comment

We thank the reviewer and appreciate that he recognizes the practical value, breadth, and the explanatory analyses in the paper. We address the reviewer questions and concerns below:

W1. Label efficiency. The linear probe is part of the evaluation, not the adaptation pipeline. Each method uses only the calibration set (N images) to adapt the backbone, then a linear head is trained on top of that frozen backbone to measure how good its representation is. The evaluation protocol is identical for all methods: same data, same schedule, same head training, so accuracy differences reflect only the quality of the backbone each method produced from the N calibration images. We have made this clearer in the experiments section. That being said, we agree with the reviewer that it is interesting to see what happens when the head is also trained on few data: using only the N calibration images for both adaptation and evaluation, Remix K=5 75.9% vs. LoRA r=4 71.9% at ViT-B/N=512 (p=0.001). Same conclusions.

W2. Failures of some baselines. The reviewer is right, the paper attributes these failures to over-parameterization but does not show the evidence.

(i) Dense Remix fits ~70M coefficients from 512 images. We now report train vs. holdout accuracy showing clear overfitting: Aircraft 98.7%(train) / 45.0% (test), CIFAR-100 91.0% / 45.5%, Caltech 96.3% / 48.0%, DTD 85.8% / 43.0%, the model memorizes the calibration set. Importantly, the same dense formulation trained on full task data reaches 75.8% vs. 76.8% for fine-tuning, so the projection itself works; it is the ratio of parameters to data that breaks it, which is exactly what sparse remixing (135K coefficients) avoids.

(ii) Ridge fits each layer in closed form with a fixed λ=1e-3. We add a regularization sweep to verify this is not just a poor hyperparameter choice (App B.3).

(iii) Neuron Merging is data-free. Its failure is not about insufficient data but about what similarity-based merging can express: it replaces each removed neuron with its most similar survivor, which cannot represent negative contributions or task-specific recombinations. The −35.3 % loss reflects this gap. We now state the mechanism behind each failure explicitly rather than grouping them under one explanation.

W3. Search allocations time. The pruning search is shared across all adaptation methods and is not part of the per-method comparison. We also ablate different pruning and search techniques (App. C.8). The wall times in App. C.5 compare the post-pruning adaptation step, since all methods start from the same pruned checkpoint, this is the fair way for comparing recovery techniques against each other. We now also reports search allocation time separately (App. C.5).

W4. Detection and segmentation. We focus on recognition tasks to systematically evaluate the core hypothesis of Channel-Remix, whether sparse recombination of pretrained channels enables data-efficient knowledge transfer, across diverse domains and pretraining families under a consistent evaluation protocol. Extending the method to detection and segmentation would require additional task-specific architectures and evaluation protocols, which we leave for future work as noted in Sec. 6.

W5. Related work. We thank the reviewer for the references, we reviewed all ten and cited relevant ones. [1, 3, 4, 8] are low-rank and adapter variants for fine-tuning pretrained ViTs; we already compare against six methods in this family (LoRA, DoRA, PiSSA, SSF, SVFT, AdaptFormer). [5] is a benchmark suite rather than a method, which we now reference. [2, 6, 7, 9, 10] address dense prediction or domain-specific settings (remote sensing, segmentation) outside our scope. We appreciate the pointers and have updated the related work section.

Reviewer_CLimOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

How to better distill a large vision foundation model into a small, compact model while keeping good performance is always an interesting research topic, and it would gain a lot of attention. In this work, the authors propose Channel-Remix, which selects particular channels as anchors and combines the remaining channels as donors. In a sense, the proposed method can be a bridge between structured channel pruning methods (hard channel selection) and learned channel projection methods (dense channel mixing), as shown in Fig. 1. Extensive experimental results demonstrate improvements.

Critical strengths
  1. The study of pruning vision foundation model is necessary and should not be ignored in these days.
  2. While I have some questions about the experiments, but the extensive experiments and settings are appreciated.
  3. Experimental results show improvements.
Critical weaknesses
  1. I would like to highlight that this submission is really not that easy to follow, and not well presented. Authors may need to reconsider how to better present the main motivation, main novelty in this work, and main contribution. Fig 1 makes the main idea in the work more like a simple combination of (a) Structured channel pruning and (b) Learned channel projection, without strong motivation and analysis behind. Also, the listed contributions (items) in the end of Sec 1 Introduction cannot hlighlight the contributions. As a demostration, these items "A foundational role for VFMs beyond supervision", "Task-specific subspace discovery via sparse recombination.", and "Comprehensive evaluation across tasks, models, calibration sizes" can be placed in most papers. From a reviewer and reader's prespective, I would kindly suggest authors could better present the work.

  2. In terms of methodology / novelty, I feel like it is little incremental. Main idea in this work is show in Fig.1 (c) and Fig. 2, select base channels and important discarded channels, weighted merge selected discarded channels to base channels. First, it is like a simple combination of Fig 1 (a) and (b) without strong motivations. Second, many works are presenting similar ideas, like following papers

  • [1] Towards meta-pruning via optimal transport

  • [2] Forget the Data and Fine-Tuning! Just Fold the Network to Compress

  • [3] Neuron Merging: Compensating for Pruned Neurons

  1. I feel like the experimental results did not prove the true effectiveness of proposed re-mixing. Like the donor weight is close to zero, and almost half are used, making the learning matrix really big, and the "data efficiency" claim is not supported.
Questions for authors
  1. I feel like Figure 3 is more like AI generated figure, like from Cladue or GPT. While it is not a weakness, I would suggest better revise it to align with the whole paper style.

  2. In Table 2, why top-grad performance is even slightly worse than random?

  3. Is it fair to compare with LORA or other PEFT methods? Proposed method can update all layers, while LORA can only adjust particular layers. If we consider LORA on all MLPs, what the performance would be? and LORA-4 is it a weak baseline?

  4. Seems experiments in C.4 showing within-head remixing is not helpful?

  5. I could not get why optimize sepecific small tasks individually.

3 comments

We thank the reviewer for the careful reading and the suggestions, which helped us make the paper clearer. Below, we address the concerns and indicate the updates.

W1. Motivation and contributions. In the Introduction, we described Channel-Remix in familiar terms so readers could grasp it easily. We agree this made it read more as a description than a motivation, and the contribution list does not make the specifics visible.

  • Why not just (a)+(b). Both fail in practice. the updated Fig. 1 show that clearly. Structured pruning (a) loses accuracy with low data. Dense remixing (b) is data-hungry and prone to overfitting. Also, our contribution is not simply sparsifying the dense map, but how: learning which donors to select per anchor and how to weight them (§3.4). We confirm this by showing in Fig.1 that sparse map + learning coefficients is well below our method, especially in low data regimes ( up to 8% gap).

We updated the introduction to reflect this motivation, Fig. 1 to make the failures explicit, and the contributions to reflect the specifics of our method.

W2. We agree the idea itself is simple, and we see that as a positive aspect. However, we respectfully disagree that it is incremental. As discussed in W1, both structured pruning and dense remixing fails. The cited works differ from ours:

  • Different mechanisms. [1, 2, 3] share one core idea: merging similar channels, with no task data and no learning. Channel-Remix instead learns what channels to combine and how to combine them per task, from a small calibration set. This matters because channels with similar weights can serve different functions [4]. Recovering a removed function may require combining dissimilar channels, sometimes with negative coefficients, which similarity-based merging cannot express.

  • Theoretical and practical evidence. App. A.8 bounds how much accuracy recombination recovers and how much is lost by using K donors instead of all channels. Such bounds cannot be derived for similarity-based merging, since it uses no task data and no learning. This shows in practice: Neuron Merging [3] loses 35.3% on average (Tab. 1).

We revised the Introduction and Related Work to state these arguments more clearly. [4] Not all solutions are created equal, Braun et al, ICML 2025.

W3. Is remixing effective? The reviewer raises a good point on the coefficients being small (~0.03), looking negligible. However, each coefficient multiplies an entire weight row, so even a 0.03-scale donor shifts the matrix by ~9.2% (L2 relative distance ) on average. Over thousand channels and a sequence of layers, these shifts accumulate into a meaningfully different representation. Two tests confirm the importance of coefficients:

  • Zeroing all donor contributions in trained models drops accuracy by 5.1% on ViT-B and 2.8% on ViT-H+ on average.
  • Training with per-channel scaling only (coefficients on anchors but not on donors) loses 17.3% of the 28.5% recovered by Remix (Table 2a).

To clarify the "half used": the number of recruited donors does not change the number of learned coefficients (size of matrix) which is lower than LoRA variants, and all are folded into the compact model after training (Sec. 3.5).

Data efficiency is supported by Fig. 4 (also the updated Fig. 1): Shows that LoRA needs 3-4x more data to match our performance.

Q1. We have updated Fig. 3 to make it clearer.

Q2. Top-gradient donors are correlated, they push in the same direction, so adding more contributes little new. Random donors are more diverse, which tend to matter more than similarity (consistent with W2). The final K are still gradient-selected by successive halving (32→16→8→K) from the initial random pool (Sec. 3.4).

Q3. LoRA is already applied to the same layers as Remix. We evaluated ranks 2, 4, and 8 because their parameter counts close to ours. We have additionally tested LoRA-16 (590K params, 4× ours Remix-K=5 ): Remix wins 74.5% vs. 72.9%. Increasing rank adds parameters but not enough data to learn them.

Q4. C.4 shows one task failure (CIFAR-100, ViT-L/100M) where attention remixing collapses. That failure is detectable from the calibration holdout, which is what motivated the safeguard (Sec. 3.4): it disables the affected layers when triggered. We updated C.4 with the full comparison: Attention+MLP remix adds +1.05 % over MLP-only. with the safeguard, this rises to +2.1%.

Q5. The fact that a task-specific compact model can be created from a VFM within minutes and with few images is what makes this setting practically valuable. This is why we focus on data and compute efficiency. One practical scenario: a large VFM sits in the cloud, and when a specific task arises (a specific domain, a constrained edge device) you extract a compact model tailored to that task and run it locally. When the task or budget changes, you rebuild it fast again from the same VFM. Remix enables this workflow under 5 min on one GPU (App. C.5).

Reviewer_CLimOpenReview ↗

Thanks for your rebuttal

Dear authors,

Thanks for your rebuttal and additional analysis. However, the rebuttal did help solve my concerns.

  1. The novelty still seems incremental to me, and the ablations seems further demonstrate that learning the donors does not matter much. Meanwhile, I dont think compare with LoRA in abstract is reasonable. I also agree that simple can be a positive aspect, but not all simple methods are good. The conditions behind are solid analysis, soild motivation, and strong performance, etc.

  2. I also have concerns on the practical value. For DINOv3, the ViT-S+ (29M) already reaches 85.0% with just a linear probe, even better than ViT-B pruned to 35M with Remix (83.2%, Table B.8). Why not just use the smaller model? Notice that ViT-S is also a distilled model from big model, and can be a baseline for this paper.

  3. The baseline comparisons seems not fair. For example, LoRA only includes two projections, while Remix changes all of them.

Based on the review and the rebuttal, I will keep my original score. Hope my comments and reviews can really help further improve this work and presentation.

Besides, thanks for pointing out the theory in A.8.  

I list this here because it seems generated by AI. I'm not considering this as a weakness and there is no impact to my rating. Just kindly reminder that some reviewer may not like this kind of sections that very similar to AI-generarted. Sometimes, such named theory proof cannot help the paper. 

All my reviews and comments are from myself without AI,  besides checking this section with AI and seems this section doesn’t really explain why the method works or why it needs less data.

We thank the reviewer for the follow-up. We are glad that our response clarified some of the concerns, and we address the remaining ones below.

1. "Learning the donors does not matter much "

The reviewer may have confused two things:

  1. Table 2(a), random donor pool: the initial 32 candidates are random, but the search still learns which K to keep from those and the coefficients (Algorithm 1 Sec 3.5 ).
  2. Random K selection: K random-based donors are chosen randomly and only the coefficients are fitted. This removes donor learning (search), which is an important part of our method and we confirm that below.

Both are tested below (ViT-B → 35M, mean over 6 tasks):

N641282565122K
Remix random pool + learned donors65.379.279.883.484.7
Remix similarity pool + learned donors68.678.479.183.183.8
Random K donors (no search)58.873.475.979.883.6
Pruning only50.666.166.371.071.4

Table 2(a) randomizes the candidate pool; the search still learns which donors to keep. It therefore shows that the pool selection rule has a small effect, not that donor learning is unnecessary. When the search is removed, accuracy drops by up to 6.5% on average and up to 15.5 on a single task (Caltech-256, N=128).

we hope our clarification addresses the concern about incremental novelty. Remix is clearly ahead of every baseline we tested: all-matrix LoRA-8/16, PiSSA, DoRA, dense projection, fixed random and similarity donors, Neuron Merging, Intra-Fusion and block reconstruction. If the reviewer thinks another baseline should be tested, we are happy to run it report the result.

2. "LoRA on all matrices"

As explained, the LoRA placement in the paper was chosen to match Remix in trainable parameter count. LoRA on all matrices of all layers (q, k, v, o, up, down) has 6–12× more trainable parameters. We tested it below with the same setting, and even with this advantage Remix wins: better than both LoRA-16 and LoRA-8 in 23/24 settings. Mean over 8 tasks:

Method (trainable params)N=128N=256N=512
Remix (135K)77.879.582.7
LoRA-16, all matrices (1.6M)76.178.081.5
LoRA-8, all matrices (0.8M)75.478.181.2
LoRA-8, (as in the paper)72.175.579.8

3. Distilled models vs Remix

The reviewer focuses on one specific setting (~30M). But the same argument works in our favor at a larger scale: The compressed H+ (841->300) beats the distilled ViT-L (304) of the same size (results below in a). It is worth noting that each distilled model is trained on ~1.7B images from a 7B teacher, while Remix is created from at most 512 images in minutes on one GPU. There are also many practical settings where remixing is clearly favorite over distillation:

(a) Larger scales (16 tasks, N=512). The compressed H+ beats the distilled ViT-L of the same size and matches the full 841M model (88.2), and also the in-between 150M is 2.7% ahead of ViT-B:

TargetBest releasedRemix
~300MViT-L (304M): 87.9H+ → 300M: 88.3
~150MViT-B 87M: 85.0L → 150M: 87.7

(b) Model Families. Of 7 families, only DINOv3 and AugReg have a model below 86M. Remix at N=512: MetaCLIP 2 +6.8, MAE +8.1, SigLIP 2 +3.9, I-JEPA +3.7, BiomedCLIP +3.4.

(c) Domain specific Foundation models. most of these models comes in one size, for example: RAD-DINO has a single release (86.6M) (DINO for radiology). Remixed to 35M it matches the full model and beats ViT-S+ by ~8 %:

TaskRAD-DINO (86.6M)ViT-S+ (29M)Remix 35M from RAD-DINO, N=512N=128
PneumoniaMNIST97.887.896.296.5
RSNA73.166.974.374.9
CheXpert-563.355.764.563.9
Mean78.170.178.378.4

(d) VLMs (new). To strength our point, We extended Remix to Larger models and more challenging tasks. Qwen2.5-VL has no release below 3B; we compressed 3.75B → 2.35B and compared with LoRA-8 over 20 tasks:

Labeled ImagesRemixLoRA-8Remix wins
10079.178.014/20
1,00084.883.920/20

4. Appendix A.8

We agree A.8 does not explain the data efficiency, and we do not claim that. It is a supporting note, which is why it is in the appendix. It only gives the motivation for learning the donors. The data-efficiency claim is empirically validated in our experiments.

We appreciate the reviewer’s effort taken in writing the reviews personally. Likewise, all presented arguments in the paper and here are our own. We are happy to answer any further concerns and to run any additional experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Summary of Revisions and Discussion

We thank the AC and all reviewers for their time and constructive feedback. Below is a summary of our responses and revisions made to the paper:

Revisions to the paper:

  • Introduction and Fig. 1: Rewrote the motivation to explain why neither structured pruning (accuracy loss at low data) nor dense remixing (data-hungry, overfits) works alone. Fig. 1 now shows both failures, plus a sparse map + learned coefficients variant that trails Channel-Remix by up to 8% at low data. The contribution list now states what is specific to our method.
  • Related Work: Clarified how we differ from similarity-based merging (Neuron Merging, folding, OT meta-pruning) and added vision PEFT references (incl. V-PETL Bench).
  • Fig. 3: Revised for clarity and consistency with the rest of the paper.
  • Experiments: Clarified that the linear probe is used only for evaluation and is identical for every method. Each baseline failure is now explained by its own mechanism, with train/holdout accuracy showing that Dense Remix overfits (e.g., Aircraft 98.7% train vs. 45.0% test).

Answers to reviewers' concerns:

Reviewer Jxn2 (calibration data, KD, cost).

  • "whether this small calibration set is in fact more important than channel-remix" → All methods share the same calibration images, teacher, KD losses and schedule, so the gains come from the method, not the data: Evaluated on 16 tasks across 6 models, Remix outperforms LoRA-8 in 85 of the 96 runs.
  • "unclear whether the distribution of this calibration set is actually balanced" → The gap is stable across calibration seeds (Remix 87.0/86.7/87.0% vs. LoRA-4 85.0/85.6/85.6%), and calibration sets are class-balanced.
  • "Is there a table comparing the training cost" → Training cost is on par with LoRA (~3.8 min). The reviewer confirmed concerns were resolved.

Reviewer nivD (evaluation protocol, baselines, scope).

  • "The final linear probes use full task training data" → The linear probe is evaluation only. Conclusions hold when the head is trained only on the NN calibration images (75.9% vs. 71.9%, p=0.001p=0.001).
  • "Severe failures of Dense Remix, Ridge, and Neuron Merging warrant stronger diagnostics" → Reasons for baseline failures are explained in the revised version, and search time is reported separately.

Reviewer CLim (novelty, donors, baselines, practical value).

  • Novelty ("it is little incremental"): Channel-Remix learns which donors to combine and how, including negative coefficients. Similarity-based merging cannot express this and loses 35.3%.
  • Donor search matters ("learning the donors does not matter much"): Table 2(a) randomizes only the candidate pool, while the search still selects the donors. Removing the search drops accuracy by up to 6.5% on average (15.5% on one task), and zeroing the donors in trained models loses 5.1% (ViT-B) and 2.8% (ViT-H+).
  • Fairness to LoRA ("LoRA only includes two projections"): LoRA was parameter-matched. Remix (135K params) still beats LoRA-8/16 applied to all matrices (0.8M–1.6M params) in 23/24 settings, and beats LoRA-16 in the paper's setting (74.5% vs. 72.9%).
  • Practical value ("Why not just use the smaller model?"): H+ → 300M (88.3%) beats the distilled ViT-L (87.9%), and L → 150M beats ViT-B by 2.7%, using ≤512 images and minutes on one GPU. On RAD-DINO, which has only one released size, Remix to 35M matches the full model and is ~8% ahead of ViT-S+. The method also scales to VLMs: compressing Qwen2.5-VL 3.75B → 2.35B, Remix beats LoRA-8 on 20/20 tasks with 1,000 labeled images.
  • A.8 ("doesn't really explain why the method works or why it needs less data"): This is a supporting note on donor learning. We do not claim it explains data efficiency, which is shown empirically.

We believe the revisions address the main concerns and thank the reviewers again for helping improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.