- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Channel-Remix uses a vision foundation model as both a teacher and a source of pretrained channels. After structured pruning, it uses a small calibration set to sparsely mix discarded donor channels into retained anchor channels, and then folds these mixtures into standard weights without adding inference cost.
Critical strengths
The authors claim that constraining adaptation to a sparse span of frozen VFM channels and folding them into a pruned backbone gives a cleaner inductive bias for scarce-data transfer than reducing parameter count alone, with no runtime overhead. They also state that the evaluation is broad and that, under matched low-data schedules, sparse remixing (K=3/5/7) is stable where dense mixing collapses and generally outperforms LoRA/PiSSA.
Critical weaknesses
Already stated in the Questions.
Questions for authors
- In lines 92-93, the authors describe “A small calibration set.” Could the authors clarify whether this small calibration set is in fact more important than channel-remix? In addition, please explain the rationale for selecting the specific calibration sets in Table B.1.
- The authors state in the appendix that these calibration sets are class-balanced. However, it is unclear whether the distribution of this calibration set is actually balanced. If it is, how was this balance determined or verified? If the distribution is inherently skewed, how can the generalization ability of the resulting small, compact model after transfer be guaranteed? In addition, would additional task data be needed to continue SFT on this compact model? I would appreciate clarification on these points.
- How large is the impact of ? Is it possible that the good performance in this experiment stems more from KD? In Table 2, does “teacher guidance” refer to the KD loss term? Clarification on these points would be appreciated.
- Is there a table comparing the training cost with that of other methods?
3 comments
Most of my questions have been resolved, and I will maintain my rating.
We thank the reviewer for the follow-up, and for the time and effort put into reviewing our paper. We are glad that our responses resolved most of the questions.
If any point remains open, or if there is any additional analysis or clarification that would make the paper a clear accept in the reviewer’s view, we would be happy to provide it.
We thank the reviewer and appreciate that he recognizes the main intuition and motivation of the paper and also values the breadth of the evaluation. We answer the reviewer questions below:
Q1. Is the calibration set more important than Channel-Remix? The reviewer raises a very good point. With so few images, it is natural to wonder how much of the gain comes from the data versus the method. Every method in our comparison uses the same calibration images, compressed model, teacher, and schedule, so accuracy differences reflect the adaptation method, not the data. Two results confirm this:
Channel-Remix beats LoRA-8 in 85/96 matched pairs (p<1e-12) with the same 512 images, and matches LoRA trained on 3–4× more data (Fig. 4).
Across three calibration draws (seeds 17, 42, 91) on ViT-B/35M, 12 tasks: Remix 87.0 / 86.7 / 87.0%, LoRA-4 85.0 / 85.6 / 85.6%. both are stable and the gap holds each time.
To clarify, Table B.1 lists the full dataset statistics; the calibration sets are class-balanced subsets drawn from the training split, not separate curated sets. The 16 datasets were chosen for domain diversity so we can confirm the method is not restricted to one domain and analyze how donor selection changes across domains, as in the cross-task analysis (Sec. 5).
Q2. Class balance, generalization, and further fine-tuning. Studying the effect of calibration set composition is an interesting direction. To answer the reviewer's specific questions:
Is it actually balanced? Yes, the sampler gives each class the same number of images (e.g., CIFAR-10 at N=512: 51–52 per class; DTD: 10–11 across 47 classes). We record the per-class counts in every run and now include them in the appendix.
Why equal per class? Simply to make sure every class has at least a few images. Random draws could miss some classes entirely, which would hurt the performance on those. Also, in our setting, we assume only a small set of images is available. Following the training set distribution would require knowing it in advance, which could be seen as using information from the full dataset. Equal sampling avoids this and ensures every class is represented.
Generalization? We did not test skewed sampling because it requires knowing the training distribution, which in our setting we assume we do not have. What we can confirm is that the method is stable across different balanced calibration sets (seeds 17/42/91: Remix 87.0/86.7/87.0%, LoRA-4 85.0/85.6/85.6% on ViT-B/35M, 12 tasks), and generalization is measured on the test sets of all 16 datasets, which are never used during adaptation.
Q3. Role of KD: Yes, "teacher guidance" in Table 2 refers to the KD and feature-matching losses. The reviewer is indeed right that KD helps, with only a few hundred images, per-layer output and representation matching prevents the compact model from overfitting. But we made sure its effect is controlled for a fair comparison:
Q4. Training cost comparison. We reported the training cost in App C.5. We did not highlight it because all methods have similar cost, it is mainly dominated by forward passes through the model. At 40 epochs on ViT-B/35M with 512 images (single GPU): LoRA-4 ~3.6 min, LoRA-8 ~3.8 min, full fine-tuning ~4.6 min, PiSSA-8 ~4.0 min, Channel-Remix ~3.8 min. Where cost really differs is data: to match Remix at 512 images, LoRA needs more data (Fig. 4), which scales training time proportionally (LoRA at 2048 images take ~15 min). Channel-Remix achieves the same accuracy in ~4 min.