- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BiSSLight, an improvement over BiSSL mainly in terms of efficiency, to address the misalignment between self-supervised pretrained representations and downstream tasks, which can degrade model accuracy and adaptation efficiency during fine-tuing. BiSSLight uses an intermediate bilevel optimization stage that couples the self-supervised pretext and downstream objectives with efficient hypergradient approximation to better align model initialization with downstream tasks before PEFT methods are applied. It shows substantial overhead improvement over BiSSL and higher accuracy over direct PEFT (LoRA).
Critical strengths
- The paper is well-written with clear presentation of the problem setup, experimental details, result analysis, and attention visualization.
- The paper studies an important problem faced by widely-used fine-tuning strategies — the training effectiveness and efficiency due to misalignment between pre-training and downstream tasks. The proposed improvement over BiSSL is interesting with using M-FAC for faster hypergraident approximation and using recycled pretext gradient.
- The original BiSSL paper also examines the field of object detection and semantic segmentation, it would be promising for the future work to show if BiSSLight extends to those fields as well.
Critical weaknesses
Some of the other concerns are in the Questions section.
- Even though the improvement on BiSSL is interesting, the paper shows limited evidence isolating the effect/importance of each used components. For example, the core proposed approximation technique is M-FAC, what about other simple approximations (e.g., diagonal Fisher approximation or truncated Neumann series) -- do they show similar improvement as M-FAC?
- Limited comparisons with other adaptation strategies. The paper only compares the proposed additional step with peft and plain peft itself, but not with other works such as ExPLoRA mentioned in the related work. The direct comparison would make it more convincing to show the effectiveness of the proposed optimization step.
Questions for authors
Thanks for submitting to ICLR 2027! The paper studies an interesting and important problem in fine-tuning / model adaptation. Here are a few questions that I think can help improve the current version of the paper (in addition to the points in the previous section).
- Here are a couple of related questions for the core design. What if comparing BiSSLight against traditional PEFT but with an equivalent total compute budget (more fine-tuning steps and updates)? That would make the comparison more fair since BiSSLight requires more upper and lower-level updates. And how does it compare against a combination of continued pre-training and fine-tuning instead of using the bilevel coupling optimization? Also, what is the compute differences between BiSSLight and PEFT? The paper does show that a big improvement from BiSSLight from BiSSL but not end-to-end comparison with BiSSLight and PEFT in terms of compute (but only in terms of accuracy). These questions can also help answer if the accuracy gain comes from extra optimization steps or the proposed bilevel coupling optimization.
- How much does BiSSLight require access to the original pre-trained datasets/unlabeled data or its objective?
5 comments
Response to Reviewer wmcB [2/2]
Continued Pre-Training
We also compare against continued MAE pre-training followed by PEFT, matched to BiSSLight's compute through multiple different combinations: pre-training steps + PEFT epochs (Ext.PT_L + PEFT); pre-training steps + PEFT extended to also cover the upper-level steps ( PEFT epochs) (Ext.PT_L + PEFT (CM_U)); and total pre-training steps + PEFT epochs (Ext.PT_{L+U} + PEFT).
| Dataset | Pets | DTD | VOC07 | Flowers | CUB200 | Aircrafts |
|---|---|---|---|---|---|---|
| PEFT | ||||||
| Ext.PT${}_L$ + PEFT | ||||||
| Ext.PT${}_L$ + PEFT (CM${}_U$) | ||||||
| Ext.PT${}_{L+U}$ + PEFT | ||||||
| BiSSLight + PEFT |
Continued pre-training rarely improves over plain PEFT and often degrades it, and none of the variants matches BiSSLight, except the single instance at Aircrafts under Ext. PT (2). Together with the compute-matched PEFT results, this indicates that the bilevel coupling itself provides gains in general. We have included all compute-matched baseline results in Section 4.4 + Table 2 of the main body.
Dependence on Pretext Data and Objective
In the default setting, BiSSLight uses the full original pre-training dataset, the pretext objective, and the pretrained pretext head. However, full access is not actually required, as one can use only a small subset of the pretext data and also fit a new pretext head for a relatively minor cost. To assess wether BiSSLight can still be beneficial under such conditions, we ran experiments using only of the unlabeled pretext data and by replacing the pretrained pretext head with a randomly initialized head that is trained with a -step warmup on the frozen backbone prior to BiSSLight.
| Dataset | Pets | DTD | VOC07 | Flowers | CUB200 | Aircrafts |
|---|---|---|---|---|---|---|
| PEFT | ||||||
| BiSSLight ( PT Data) + PEFT | ||||||
| BiSSLight (random pretext head + 1k warmup) + PEFT | ||||||
| BiSSLight + PEFT |
Both restricted settings match the default BiSSLight+PEFT rather well and still clearly outperform PEFT in general. Thus BiSSLight requires a pretext objective and some unlabeled data, but neither the full original pre-training dataset nor the pretrained pretext head is strictly necessary. The results are reported in Section D.3 of the paper.
Dear Reviewer. We have responded to your constructive review, incorporating additional experiments into the revised maniscript to address the concerns raised. As the end of the discussion period comes closer, we kindly ask you to consider our rebuttal response. We are happy to provide further clarifications or discuss any further questions you may have. Thank you.
Response to Reviewer wmcB [1/2]
Thank you for taking the time to read our paper and making a constructive review. We appreciate that the presentation of the paper is considered clear and well-written and that our proposed approach is found interesting. Below we address the concerns. All new experiments will be added to the paper, and the pdf will soon be updated accordingly.
Contribution from M-FAC
We agree that a more close examination of the contribution from M-FAC is reasonable, and we have implemented a diagonal-Fisher variant of BiSSLight. This variant keeps only the diagonal of the empirical Fisher, D=(1/NL)∑igi⊙gi, giving the element-wise inverse approximation (D+λI)−1. Its similarly constructed from recycled lower-level pretext gradients, exactly as M-FAC does. A truncated Neumann series shares CG's bottleneck of repeated Hessian-vector products, so the diagonal Fisher is the cheaper simple alternative to compare against. We use the main experimental setup with the ViT-L backbone, as described in the paper.
The diagonal Fisher matches M-FAC quite well, but generally lands a little below it, with a larger gap on Flowers (+0.5). We hypothesize that this is attributed by M-FAC being able to capture second-order interactions between parameters, which a diagonal approximation cannot represent. The diagonal variant is still a reasonable, even lighter alternative when storing the NL lower-level gradients is not feasible. Computationally, however, the two are practically the same, since the extra M-FAC operations are only a few dot products over the trainable parameters. M-FAC therefore gives the best accuracy in the PEFT scenarios of the paper at no practical extra cost. These results are now included in Section D.2.2 and Table 7 of the paper.
Extended Baseline Comparisons
Compute-Comparison
Recall NL=20 and NU=8 are the number of lower and upper-level iterations, while T=500 is the number of training stage alternations in BiSSLight. BiSSLight adds a total of (NU+NL)T=(8+20)⋅500=14,000 optimization steps on top of the 200-epoch PEFT. In wall-clock time, 200-epoch PEFT takes roughly 0.5−1.5 (ViT-B), 1−2 (ViT-L), and 2−6 (ViT-H) GPU Hours depending on the downstream dataset size, while the BiSSLight stage adds a fixed 4, 5, and 8 GPU hours respectively. We note these compute times in the paper (Section D.4.2.).
Compute-Matched PEFT Baseline
To re-emphasize, BiSSLight performs a total of (NU+NL)T=14,000 upper/lower-level steps, which is then followed by 200 epochs of subsequent PEFT. To test whether the gains come from extra compute alone, we extend the PEFT baseline to the same total number of optimization steps, i.e., 200+14,000/(N/B) epochs, where B=256 is the batch size and N is the size of the downstream dataset (corresponding to a total of 947-3,713 epochs depending on dataset size). We again use the paper default experimental setup on ViT-L. Note that this baseline is generous in runtime as well, as extended PEFT for ViT-L runs 9−12 GPU hours, which is more than the roughly 6−7 GPU hours of BiSSLight + PEFT in total.
Extended PEFT improves on the Flowers and Aircrafts datasets but is neutral or worse on the remaining datasets, and BiSSLight + PEFT attain the best accuracy on all six datasets, with clear margins on DTD, VOC07, CUB200 and Aircrafts (+0.9, +0.6, +1.3, +1.2). This implies that the gain is therefore not explained by additional optimization steps alone.
Continued...