acceptodds
Under review as a conference paper at ICLR 2027

A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task‑Specific Learning

Abstract

Self-supervised pretraining learns representations that are broadly transferable across downstream tasks, yet direct fine-tuning can be suboptimal due to misalignment between self-supervised and downstream task objectives, potentially degrading pretrained features beneficial to the downstream task. The BiSSL framework addressed this by introducing a transitional training stage formulated as a bilevel optimization problem, in which the downstream task objective guides the self-supervised learning process in refining pretrained representations to better facilitate subsequent fine-tuning. However, BiSSL relies on conventional bilevel optimization solving techniques whose costly implicit hypergradient approximations render the method increasingly impractical for contemporary model architectures. To make it efficient and scalable, we introduce BiSSLight, which combines M‑FAC‑based implicit gradient approximation with parameter‑efficient fine‑tuning via LoRA, enabling efficient application at larger scales that were previously impractical. Evaluation across multiple downstream tasks and contemporary model architectures shows that BiSSLight consistently improves downstream performance, with gains becoming more pronounced as model size increases despite stronger baselines. The method is highly computationally efficient, reducing computation time by more than a factor of ten compared to its predecessor on a ViT-H backbone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Fine-Tuning, Parameter-Efficient Fine-Tuning, Transfer Learning, Self-Supervised Learning, Bilevel Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wmcBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BiSSLight, an improvement over BiSSL mainly in terms of efficiency, to address the misalignment between self-supervised pretrained representations and downstream tasks, which can degrade model accuracy and adaptation efficiency during fine-tuing. BiSSLight uses an intermediate bilevel optimization stage that couples the self-supervised pretext and downstream objectives with efficient hypergradient approximation to better align model initialization with downstream tasks before PEFT methods are applied. It shows substantial overhead improvement over BiSSL and higher accuracy over direct PEFT (LoRA).

Critical strengths
  • The paper is well-written with clear presentation of the problem setup, experimental details, result analysis, and attention visualization.
  • The paper studies an important problem faced by widely-used fine-tuning strategies — the training effectiveness and efficiency due to misalignment between pre-training and downstream tasks. The proposed improvement over BiSSL is interesting with using M-FAC for faster hypergraident approximation and using recycled pretext gradient.
  • The original BiSSL paper also examines the field of object detection and semantic segmentation, it would be promising for the future work to show if BiSSLight extends to those fields as well.
Critical weaknesses

Some of the other concerns are in the Questions section.

  • Even though the improvement on BiSSL is interesting, the paper shows limited evidence isolating the effect/importance of each used components. For example, the core proposed approximation technique is M-FAC, what about other simple approximations (e.g., diagonal Fisher approximation or truncated Neumann series) -- do they show similar improvement as M-FAC?
  • Limited comparisons with other adaptation strategies. The paper only compares the proposed additional step with peft and plain peft itself, but not with other works such as ExPLoRA mentioned in the related work. The direct comparison would make it more convincing to show the effectiveness of the proposed optimization step.
Questions for authors

Thanks for submitting to ICLR 2027! The paper studies an interesting and important problem in fine-tuning / model adaptation. Here are a few questions that I think can help improve the current version of the paper (in addition to the points in the previous section).

  • Here are a couple of related questions for the core design. What if comparing BiSSLight against traditional PEFT but with an equivalent total compute budget (more fine-tuning steps and updates)? That would make the comparison more fair since BiSSLight requires more upper and lower-level updates. And how does it compare against a combination of continued pre-training and fine-tuning instead of using the bilevel coupling optimization? Also, what is the compute differences between BiSSLight and PEFT? The paper does show that a big improvement from BiSSLight from BiSSL but not end-to-end comparison with BiSSLight and PEFT in terms of compute (but only in terms of accuracy). These questions can also help answer if the accuracy gain comes from extra optimization steps or the proposed bilevel coupling optimization.
  • How much does BiSSLight require access to the original pre-trained datasets/unlabeled data or its objective?
5 comments
AuthorsOpenReview ↗

Response to Reviewer wmcB [1/2]

Thank you for taking the time to read our paper and making a constructive review. We appreciate that the presentation of the paper is considered clear and well-written and that our proposed approach is found interesting. Below we address the concerns. All new experiments will be added to the paper, and the pdf will soon be updated accordingly.

Contribution from M-FAC

We agree that a more close examination of the contribution from M-FAC is reasonable, and we have implemented a diagonal-Fisher variant of BiSSLight. This variant keeps only the diagonal of the empirical Fisher, D=(1/NL)∑igi⊙giD = (1/N_L) \sum_i g_i \odot g_i, giving the element-wise inverse approximation (D+λI)−1(D + \lambda I)^{-1}. Its similarly constructed from recycled lower-level pretext gradients, exactly as M-FAC does. A truncated Neumann series shares CG's bottleneck of repeated Hessian-vector products, so the diagonal Fisher is the cheaper simple alternative to compare against. We use the main experimental setup with the ViT-L backbone, as described in the paper.

DatasetPetsDTDVOC07FlowersCUB200Aircrafts
PEFT94.4±0.394.4\pm0.374.1±0.674.1\pm0.685.5±0.285.5\pm0.294.6±0.494.6\pm0.484.4±0.484.4\pm0.479.1±0.479.1\pm0.4
BiSSLight (Diag. Fisher) + PEFT94.7±0.394.7\pm0.374.6±0.574.6\pm0.585.8±0.285.8\pm0.295.3±0.395.3\pm0.385.5±0.285.5\pm0.282.1±0.282.1\pm0.2
BiSSLight (M-FAC) + PEFT94.7±0.294.7\pm0.274.9±0.474.9\pm0.485.9±0.185.9\pm0.195.8±0.295.8\pm0.285.6±0.385.6\pm0.382.2±0.382.2\pm0.3

The diagonal Fisher matches M-FAC quite well, but generally lands a little below it, with a larger gap on Flowers (+0.5). We hypothesize that this is attributed by M-FAC being able to capture second-order interactions between parameters, which a diagonal approximation cannot represent. The diagonal variant is still a reasonable, even lighter alternative when storing the NLN_L lower-level gradients is not feasible. Computationally, however, the two are practically the same, since the extra M-FAC operations are only a few dot products over the trainable parameters. M-FAC therefore gives the best accuracy in the PEFT scenarios of the paper at no practical extra cost. These results are now included in Section D.2.2 and Table 7 of the paper.

Extended Baseline Comparisons

Compute-Comparison

Recall NL=20N_L=20 and NU=8N_U=8 are the number of lower and upper-level iterations, while T=500T=500 is the number of training stage alternations in BiSSLight. BiSSLight adds a total of (NU+NL)T=(8+20)⋅500=14,000(N_U + N_L) T =(8 + 20)\cdot 500 = 14,000 optimization steps on top of the 200200-epoch PEFT. In wall-clock time, 200200-epoch PEFT takes roughly 0.5−1.50.5-1.5 (ViT-B), 1−21-2 (ViT-L), and 2−62-6 (ViT-H) GPU Hours depending on the downstream dataset size, while the BiSSLight stage adds a fixed 44, 55, and 88 GPU hours respectively. We note these compute times in the paper (Section D.4.2.).

Compute-Matched PEFT Baseline

To re-emphasize, BiSSLight performs a total of (NU+NL)T=14,000(N_U + N_L)T=14,000 upper/lower-level steps, which is then followed by 200200 epochs of subsequent PEFT. To test whether the gains come from extra compute alone, we extend the PEFT baseline to the same total number of optimization steps, i.e., 200+14,000/(N/B)200 + 14,000/(N/B) epochs, where B=256B=256 is the batch size and NN is the size of the downstream dataset (corresponding to a total of 947947-3,7133,713 epochs depending on dataset size). We again use the paper default experimental setup on ViT-L. Note that this baseline is generous in runtime as well, as extended PEFT for ViT-L runs 9−129-12 GPU hours, which is more than the roughly 6−76-7 GPU hours of BiSSLight + PEFT in total.

DatasetPetsDTDVOC07FlowersCUB200Aircrafts
PEFT94.4±0.394.4\pm0.374.1±0.674.1\pm0.685.5±0.285.5\pm0.294.6±0.494.6\pm0.484.4±0.484.4\pm0.479.1±0.479.1\pm0.4
PEFT (CM$_{L+U}$)94.6±0.294.6\pm0.274.0±0.674.0\pm0.685.3±0.385.3\pm0.395.7±0.295.7\pm0.284.3±0.584.3\pm0.581.0±0.381.0\pm0.3
BiSSLight + PEFT94.7±0.294.7\pm0.274.9±0.474.9\pm0.485.9±0.185.9\pm0.195.8±0.295.8\pm0.285.6±0.385.6\pm0.382.2±0.382.2\pm0.3

Extended PEFT improves on the Flowers and Aircrafts datasets but is neutral or worse on the remaining datasets, and BiSSLight + PEFT attain the best accuracy on all six datasets, with clear margins on DTD, VOC07, CUB200 and Aircrafts (+0.9, +0.6, +1.3, +1.2). This implies that the gain is therefore not explained by additional optimization steps alone.

Continued...

AuthorsOpenReview ↗

Response to Reviewer wmcB [2/2]

Continued Pre-Training

We also compare against continued MAE pre-training followed by PEFT, matched to BiSSLight's compute through multiple different combinations: NLT=20⋅500=10,000N_L T = 20 \cdot 500 = 10,000 pre-training steps + 200200 PEFT epochs (Ext.PT_L + PEFT); 10,00010,000 pre-training steps + PEFT extended to also cover the NUT=4,000N_U T = 4,000 upper-level steps (200+4,000/(N/B)200 + 4,000/(N/B) PEFT epochs) (Ext.PT_L + PEFT (CM_U)); and (NL+NU)T=14,000(N_L+N_U)T = 14,000 total pre-training steps + 200200 PEFT epochs (Ext.PT_{L+U} + PEFT).

DatasetPetsDTDVOC07FlowersCUB200Aircrafts
PEFT94.4±0.394.4\pm0.374.1±0.674.1\pm0.685.5±0.285.5\pm0.294.6±0.494.6\pm0.484.4±0.484.4\pm0.479.1±0.479.1\pm0.4
Ext.PT${}_L$ + PEFT94.0±0.394.0\pm0.374.0±0.474.0\pm0.484.8±0.284.8\pm0.294.9±0.494.9\pm0.484.7±0.384.7\pm0.381.3±0.381.3\pm0.3
Ext.PT${}_L$ + PEFT (CM${}_U$)93.9±0.493.9\pm0.473.2±0.673.2\pm0.684.7±0.284.7\pm0.295.4±0.395.4\pm0.384.3±0.484.3\pm0.482.8±0.482.8\pm0.4
Ext.PT${}_{L+U}$ + PEFT94.2±0.394.2\pm0.374.0±0.374.0\pm0.384.6±0.384.6\pm0.394.7±0.394.7\pm0.384.1±0.684.1\pm0.681.5±0.681.5\pm0.6
BiSSLight + PEFT94.7±0.294.7\pm0.274.9±0.474.9\pm0.485.9±0.185.9\pm0.195.8±0.295.8\pm0.285.6±0.385.6\pm0.382.2±0.382.2\pm0.3

Continued pre-training rarely improves over plain PEFT and often degrades it, and none of the variants matches BiSSLight, except the single instance at Aircrafts under Ext. PT (2). Together with the compute-matched PEFT results, this indicates that the bilevel coupling itself provides gains in general. We have included all compute-matched baseline results in Section 4.4 + Table 2 of the main body.

Dependence on Pretext Data and Objective

In the default setting, BiSSLight uses the full original pre-training dataset, the pretext objective, and the pretrained pretext head. However, full access is not actually required, as one can use only a small subset of the pretext data and also fit a new pretext head for a relatively minor cost. To assess wether BiSSLight can still be beneficial under such conditions, we ran experiments using only 10%10\% of the unlabeled pretext data and by replacing the pretrained pretext head with a randomly initialized head that is trained with a 1,0001,000-step warmup on the frozen backbone prior to BiSSLight.

DatasetPetsDTDVOC07FlowersCUB200Aircrafts
PEFT94.4±0.394.4\pm0.374.1±0.674.1\pm0.685.5±0.285.5\pm0.294.6±0.494.6\pm0.484.4±0.484.4\pm0.479.1±0.479.1\pm0.4
BiSSLight (10%10\% PT Data) + PEFT94.6±0.394.6\pm0.374.7±0.374.7\pm0.385.9±0.185.9\pm0.195.8±0.295.8\pm0.285.4±0.385.4\pm0.382.5±0.382.5\pm0.3
BiSSLight (random pretext head + 1k warmup) + PEFT94.9±0.194.9\pm0.174.2±0.374.2\pm0.385.9±0.285.9\pm0.295.8±0.395.8\pm0.385.6±0.385.6\pm0.382.8±0.382.8\pm0.3
BiSSLight + PEFT94.7±0.294.7\pm0.274.9±0.474.9\pm0.485.9±0.185.9\pm0.195.8±0.295.8\pm0.285.6±0.385.6\pm0.382.2±0.382.2\pm0.3

Both restricted settings match the default BiSSLight+PEFT rather well and still clearly outperform PEFT in general. Thus BiSSLight requires a pretext objective and some unlabeled data, but neither the full original pre-training dataset nor the pretrained pretext head is strictly necessary. The results are reported in Section D.3 of the paper.

Dear Reviewer. We have responded to your constructive review, incorporating additional experiments into the revised maniscript to address the concerns raised. As the end of the discussion period comes closer, we kindly ask you to consider our rebuttal response. We are happy to provide further clarifications or discuss any further questions you may have. Thank you.

Reviewer_7Rf6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BiSSLight, an efficient variant of BiSSL, which inserts a bilevel optimization stage between self-supervised pre-training and downstream fine-tuning. It replaces the conjugate-gradient solver for the inverse-Hessian-vector product with a block-wise M-FAC approximation that reuses lower-level gradients, and uses LoRA in both levels. Experiments on ViT-B/L/H with MAE (and I-JEPA) show a substantial speed-up over BiSSL and small accuracy gains over direct LoRA fine-tuning on six image classification datasets.

Critical strengths
  • The paper is clearly written and easy to follow.
  • The efficiency gain over BiSSL is substantial and makes the bilevel stage feasible for ViT-H.
  • The method yields consistent, though small, gains across model sizes and two pretext tasks.
Critical weaknesses
  • Limited technical contribution. The work is mainly an easier-to-use extension of BiSSL, combining existing tools (M-FAC and LoRA) to accelerate an extra training stage between pre-training and fine-tuning. Beyond efficiency, it brings no new insight into why or when this stage helps. It also inherits the limitations of its predecessor: it requires access to the pre-training data and the pretext task (e.g., the MAE decoder), must be rerun for each downstream task, and makes the training pipeline considerably more complex.
  • Weak evaluation makes the contribution less solid. The improvements in Table 1 are small. Many gains are only slightly larger than the standard deviation, and on Pets they are comparable to it. Given the additional effort and pipeline complexity compared with the vanilla PEFT baseline, the practical benefit is unclear.
  • Limited scope of evaluation. All experiments are on image classification, mainly fine-grained datasets. It remains unknown whether BiSSLight is effective in more complex downstream scenarios such as object detection or semantic segmentation.
Questions for authors

See weekness part.

2 comments
AuthorsOpenReview ↗

Response to Revewer 7Rf6

Thank you for the review. We are glad that the paper was found to be clearly written and easy to follow. We also appreciate the recognition that the efficiency gains over BiSSL are substantial, and that the accuracy gains are consistent across model sizes and pretext tasks, which collectively are among the central strengths of the work. Below we address the three points raised by the review.

Technical Contribution

We agree that BiSSLight inherits some limitations of BiSSL, but we respectfully disagree with two points. First, rerunning the BiSSLight alignment stage per downstream task is cheap precisely because of the paper's core contribution. The efficiency gain highlighted above makes this stage practical to run for each task. Second, while introducing any intermediate alignment stage inevitably increases pipeline complexity, in practice the additional burden is small for BiSSLight specifically. The lower- and upper-levels reuse the configurations of the existing pretext and fine-tuning stages, and only a small set of new hyperparameters is introduced. These are kept constant across all main experiments, which we show is robust. Only the upper-level learning rate is tuned per dataset.

On the reliance on pretext data and the pretrained pretext head: neither is strictly required. When the original pretext data or head is unavailable, one can use a small subset of the pretext dataset as well as one can fit a new pretext head on the frozen backbone. We verified both options for BiSSLight (also detailed in our response to Reviewer wmcB):

DatasetPetsDTDVOC07FlowersCUB200Aircrafts
PEFT94.4±0.394.4\pm0.374.1±0.674.1\pm0.685.5±0.285.5\pm0.294.6±0.494.6\pm0.484.4±0.484.4\pm0.479.1±0.479.1\pm0.4
BiSSLight (10%10\% PT Data) + PEFT94.6±0.394.6\pm0.374.7±0.374.7\pm0.385.9±0.185.9\pm0.195.8±0.295.8\pm0.285.4±0.385.4\pm0.382.5±0.382.5\pm0.3
BiSSLight (random pretext head + 1k warmup) + PEFT94.9±0.194.9\pm0.174.2±0.374.2\pm0.385.9±0.285.9\pm0.295.8±0.395.8\pm0.385.6±0.385.6\pm0.382.8±0.382.8\pm0.3
BiSSLight + PEFT94.7±0.294.7\pm0.274.9±0.474.9\pm0.485.9±0.185.9\pm0.195.8±0.295.8\pm0.285.6±0.385.6\pm0.382.2±0.382.2\pm0.3

Using only 10%10\% of the pre-training data mostly matches full-data accuracy, and a randomly initialized head with a 1,0001,000-step warm-up overall matches using the the pretrained head. The method therefore remains applicable and effective when full pretext access is unavailable.

Strength of Evaluation

We respectfully disagree that the evaluation is weak. BiSSLight + PEFT achieves a higher accuracy than vanilla PEFT on all six datasets with statistical significance in the vast majority of cases. The gains are in some cases even very prominent, e.g. for ViT-L on Aircrafts (+3.1), Flowers (+1.2), and CUB200 (+1.2), each several times the reported standard deviations. Moreover, the compute-matched baselines added during this rebuttal (see response to Reviewer wmcB) show that designing an intermediate stage that reliably helps is not straightforward. PEFT trained with a matched total compute budget or continued pre-training improves on only some datasets and is neutral or worse on others. BiSSLight is the only method that reliably improves or maintains performance across all datasets.

Scope of Evaluation

The paper covers six different downstream classification datasets, multiple model sizes, two pretext tasks, ablations over varying LoRA ranks, and the rebuttal adds limited pretext knowledge settings, compute-matched PEFT and continued-pre-training baselines and results on the approximation quality and attention alignment. We believe this substantiates the paper's claims. We agree that object detection and semantic segmentation are interesting extensions for future work (incidentally, the original BiSSL work already examined these tasks and found significant improvements), but we do not believe their absence diminishes the strength of the presented results.

Thank you once again for taking the time to review our work. As the discussion period draws to a close, we kindly ask you to take a look at our response. We are, of course, happy to continue the discussion and provide any additional clarification if needed.

Reviewer_Js5hOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces BiSSLight, i.e. a computationally cheap version of the BiSSL training algorithm, to align features learned by self-supervised learning (SSL) to a specific downstream task by solving a bi-level optimization problem. More specifically, BiSSLight combines BiSSL with M-FAC-based parameter approximations and LoRA, yielding a 10×10 \times computationally more efficient algorithm. In practice this makes BiSSLight applicable to larger models. The paper shows in experiments that bilevel refinement is particulary helpful as model size increases.

Critical strengths
  • Paper is well written and easy to read, with a clear practical contribution
  • Well motivated and solution oriented, applying well known methods (PEFT and M-FAC) to reduce memory requirement and computational cost. The way how the authors applied M-FAC is well described and understandable.
  • Experiments show consistent improvements across different settings
  • The paper provides detailed pseudo-code, which makes re-implementation straight forward.
Critical weaknesses

Benefit of bilevel coupling not isolated

BiSSLight adds substantial training before finetuning. The provided PEFT baseline never sees this kind of additional training. Hence, I would argue the comparison is not fair. A fair experimental setup would be to match the compute for pure PEFT and BiSSLight + PEFT to see which one converges to the better performance under the same computational budget.

Quality of M-FAC approximation

The M-FAC approximation is one of the core contributions of the paper. However, the paper does not directly show that this approximation is meaningful. It never directly reports how strong the approximation is empirically. The paper only reports downstream accuracy.

The justification for the empirical Fisher as a Hessian surrogate (Sections 2.2 and 3.1) relies on the model being near a pretext optimum. According to the paper, this holds because training starts from a pretrained checkpoint. However, BiSSLight moves θP\theta_P away from this point. The reported CKA of ~0.54 to the pretrained backbone (Figure 5) indicates substantial representational change.

The paper does not compare against a reference hypergradient at any point in training. It also does not include a comparison to a straight forward simplification that removes the second-order term and uses a first-order hypergradient.

Hence, it is unclear if the approximation is accurate and whether the applied curvature correction contributes anything to the performance of BiSSLight.

No analytical complexity characterization

The claims of the paper that BiSSLight scales well with model size are purely based on Figure 2. The paper provides no theoretical analysis for cost or memory requirement as a fucntion of the number of trainable parameters, LoRA rank, NLN_L, NUN_U, ...

Without such an analysis, it is unclear how much of the reported scaling advantage is algorithmic or based on experimental setup (e.g. GPU count changes from 2 to 4 in some experiments).

Attention analysis leaves room for different interpretations Section 4.4 provide attention rolloiut maps for a selected model on a handful of images. The authors draw conclusions about improved alignment. A quantitative measure is missing. It is hard to follow the claims purely based on visual interpretation.

Questions for authors
  1. Please provide asymptotic per-iteration cost and memory requirement of BiSSL and BiSSLight as a function of trainable parameters dd (depending on LoRA parameter count which scales with rr and depth), the number of CG iterations for BiSSL, NLN_L, NUN_U and the number of blocks KK.

  2. Section 4.3 states that BiSSL in Figure 2 uses the same LoRA configuration as BiSSLight. The reported 5-10x gain therefore comes only from replacing CG, not from LoRA. Can you also report full-parameter BiSS so the LoRA contribution is quantified separately?

  3. How close is the M-FAC hypergradient to a reference, for example CG with many iterations, at several points during training (start, middle, end)? Please report cosine similarity or relative error, and how it varies with LoRA rank and NLN_L.

  4. What is the total cost of the BiSSLight stage plus fine-tuning, compared with PEFT alone?

  5. The Fisher–Hessian equivalence assumes a likelihood-based loss and a well-specified model. The MAE normalized-pixel MSE loss and the I-JEPA latent-prediction loss are not obviously of this form. What justifies the empirical Fisher as a curvature surrogate here?

  6. BiSSLight adds roughly 2 ImageNet epochs of MAE training, 4,000 upper-level steps on downstream data, and a 20-epoch linear probe. Can you add compute-matched baselines, such as (a) LoRA continued MAE pretraining with no upper level (similar to ExPLoRA), (b) upper-level/downstream-only training of the same length, and (c) PEFT with an equally extended schedule? This would isolate the benefit of the bilevel coupling itself.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Js5h [1/4]

Thank you for the detailed and constructive review, which have helped strengthen the paper. We appreciate the positive assessment of the presentation, the practical contribution, the consistent improvements across settings, and the reproducibility effort. We find the questions interesting and agree with most points raised and will address them in the paper, and the pdf being updated accordingly soon. Below, we answer each question.

Q1

1: Please provide asymptotic per-iteration cost and memory requirement of BiSSL and BiSSLight as a function of trainable parameters dd (depending on LoRA parameter count that scales with rr and depth), the number of CG iterations for BiSSL CC, NLN_L, NUN_U and the number of blocks KK.

Let QQ be the cost of one forward pass and BB one backward pass. A backward pass is usually proportional to a forward pass up to a constant factor, so O(B)=O(Q)\mathcal{O}(B) = \mathcal{O}(Q). The lower-level training cost is the standard cost of training. Executing NLN_L iterations gives an asymptotic compute cost of O(NLQ)\mathcal{O}(N_LQ). The cost of the upper-level hypergradient approximation is where the methods differ, which we outline below.

BiSSL (CG): Each of the NUN_U upper-level steps runs CG for CC iterations. Each CG-iteration is dominated by one Hessian-vector product, which costs one forward and two backward passes, providing an asymptotic cost of O(NUCQ)\mathcal{O}(N_U C Q). CG inherently stores four dd-dimensional vectors, with dd being the number of trainable parameters, so its asymptotic memory complexity is O(d)\mathcal{O}(d). Including the first-order upper-level gradient, which costs one forward and one backward pass per upper-level step, the total asymptotic compute for the upper-level of BiSSL is still O(NU(CQ+Q))=O(NUCQ)\mathcal{O}(N_U(CQ + Q))=\mathcal{O}(N_UCQ), i.e. the total cost is dominated by the cost of executing CG.

BiSSLight (M-FAC): M-FACuses only elementary vector operations over the dd trainable parameters when acquiring the NLN_L M-FAC vectors and forming the hypergradient approximation. The computational cost of these elementary operations is asymptotically O(d(NL2+NUNL))\mathcal{O}(d(N_L^2 + N_U N_L)). Thus, the hypergradient approximation cost of BiSSLight only scales with the number of trainable parameters dd, not with the cost of the substantially heavier forward/backward passes. The total working memory for M-FAC is NL(d+K)+d≈(NL+1)dN_L(d + K) + d\approx (N_L +1)d (since K≪dK \ll d), resulting from the storage of the NLN_L dd-dimensional M-FAC vectors, NLKN_L K denominator scalar values, and the first-order upper-level gradient, so its asymptotic memory complexity is O(NLd)\mathcal{O}(N_Ld). Including the upper-level first-order gradient calculation, the total asymptotic compute for the upper-level of BiSSLight is O(d(NL2+NUNL)+NUQ)\mathcal{O}(d(N_L^2 + N_U N_L) + N_UQ).

The above expressions imply the cost reduction of BiSSLight over BiSSL is bounded only by a factor of CC, but we emphasize this gap is larger in practice. Looking solely at the hypergradient approximation cost (the only place they differ), BiSSL pays O(NUCQ)\mathcal{O}(N_UCQ) for CG while BiSSLight pays only O(d(NL2+NUNL))\mathcal{O}(d(N_L^2 + N_UN_L)) for M-FAC. For LoRA applied to a transformer specifically, O(d)=O(rLn)\mathcal{O}(d)=\mathcal{O}(rLn), with LL the number of hidden layers and nn the hidden dimension, so one can substitute this into the expressions above, though we keep the general form depending on dd for generality.

We have added this analysis in Appendix B.1.

Q2

2: Section 4.3 states that BiSSL in Figure 2 uses the same LoRA configuration as BiSSLight. The reported 5-10x gain therefore comes only from replacing CG, not from LoRA. Can you also report full-parameter BiSSL so the LoRA contribution is quantified separately?

To quantify the LoRA contribution separately, we have run full-parameter fine-tuning on the identical setup as used to benchmark the compute in the paper (for ViT-B and ViT-L, as our hardware did not have enough memory for ViT-H):

GPU Hours (ViT-B / ViT-L)BiSSL (CG)BiSSLight (M-FAC)
Full FT43 / 86 h9 / 44 h
LoRA22 / 37 h4 / 5 h
Peak GPU Memory (ViT-B / ViT-L)BiSSL (CG)BiSSLight (M-FAC)
Full FT74 / 165 GB20 / 73 GB
LoRA58 / 77 GB14 / 22 GB

LoRA alone reduces runtime by roughly 2x and up to 8.8x, and peak memory by 1.3-3.3x depending on the model. Replacing CG with M-FAC gives a further 2-7.4x speedup.

We note that BiSSLight is intended for light PEFT-style tuning. With full fine-tuning, dd grows large and the CPU-executed M-FAC operations become the bottleneck, explaining the notably longer runtime of 4444 GPU hours for ViT-L. We hypothesize that a variant offloading the heavier calculations to the GPU while avoiding storing all M-FAC vectors in VRAM would speed this up. Nevertheless, we state this limitation and include the above results (Table 10, Appendix D.4.2.)

Continued...

AuthorsOpenReview ↗

Response to Reviewer Js5h [2/4]

Q3

3: How close is the M-FAC hypergradient to a reference, for example CG with many iterations, at several points during training (start, middle, end)? Please report cosine similarity or relative error, and how it varies with LoRA rank and NLN_L.

We compare the M-FAC hypergradient against a strong reference computed with CG. The CG-reference uses the average HVP over the same NLN_L batches from which the precalculated gradients used by M-FAC are computed. Based on preliminary testing, a dampening of 0.10.1 and 55 CG iterations makes the reference converge well, so we use that. For varying NLN_L and LoRA rank rr, we report cosine similarity between the full M-FAC and CG hypergradients, relative error, and cosine similarity between each hypergradient with the first-order upper-level gradient subtracted. We ran these experiments with the main experimental setup for BiSSLight as specified in the paper, using the ViT-B backbone on the CUB200 dataset, and measured at the start, middle, and end of training.

NLN_L (Start / mid / end)Cosine SimilarityRelative ErrorCosine Similarity of Correction
110.99787/0.3455/0.968150.99787 / 0.3455 / 0.968150.0586/1.6982/0.24550.0586 / 1.6982 / 0.24550.3317/0.3186/0.28950.3317 / 0.3186 / 0.2895
220.99906/0.99902/0.998940.99906 / 0.99902 / 0.998940.0394/0.0443/0.04590.0394 / 0.0443 / 0.04590.5227/0.1277/0.12500.5227 / 0.1277 / 0.1250
550.99861/0.99796/0.997870.99861 / 0.99796 / 0.997870.0499/0.0614/0.06190.0499 / 0.0614 / 0.06190.5765/0.5081/0.39730.5765 / 0.5081 / 0.3973
10100.99907/0.99811/0.997260.99907 / 0.99811 / 0.997260.0410/0.0590/0.07240.0410 / 0.0590 / 0.07240.8100/0.6518/0.56070.8100 / 0.6518 / 0.5607
20\mathbf{20} (Default)0.99943/0.99898/0.998010.99943 / 0.99898 / 0.998010.0310/0.0438/0.05990.0310 / 0.0438 / 0.05990.6276/0.6893/0.65440.6276 / 0.6893 / 0.6544
30300.99922/0.99894/0.998870.99922 / 0.99894 / 0.998870.0366/0.0495/0.04670.0366 / 0.0495 / 0.04670.8374/0.7223/0.68340.8374 / 0.7223 / 0.6834
50500.99866/0.99894/0.998160.99866 / 0.99894 / 0.998160.0472/0.0442/0.05690.0472 / 0.0442 / 0.05690.7283/0.7391/0.71680.7283 / 0.7391 / 0.7168
1001000.99970/0.99923/0.997650.99970 / 0.99923 / 0.997650.0229/0.0391/0.06460.0229 / 0.0391 / 0.06460.7256/0.7444/0.80980.7256 / 0.7444 / 0.8098
Lora rank rr (Start / mid / end)Cosine SimilarityRelative ErrorCosine Similarity of Correction
110.99996/0.99971/0.999580.99996 / 0.99971 / 0.999580.0084/0.0238/0.02830.0084 / 0.0238 / 0.02830.5019/0.7358/0.48670.5019 / 0.7358 / 0.4867
220.99998/0.99986/0.999190.99998 / 0.99986 / 0.999190.0054/0.0160/0.03750.0054 / 0.0160 / 0.03750.5016/0.6699/0.67180.5016 / 0.6699 / 0.6718
880.99961/0.99963/0.999450.99961 / 0.99963 / 0.999450.0266/0.0251/0.03250.0266 / 0.0251 /0.03250.6870/0.5774/0.50660.6870 / 0.5774 / 0.5066
16160.99987/0.99959/0.999480.99987 / 0.99959 / 0.999480.01558/0.0288/0.03210.01558 / 0.0288 / 0.03210.7699/0.6799/0.58490.7699 / 0.6799 / 0.5849
64\mathbf{64} (Default)0.99943/0.99898/0.998010.99943 / 0.99898 / 0.998010.0310/0.0438/0.05990.0310 / 0.0438 / 0.05990.6276/0.6893/0.65440.6276 / 0.6893 / 0.6544
1281280.99815/0.99905/0.998480.99815 / 0.99905 / 0.998480.0569/0.0434/0.05190.0569 / 0.0434 / 0.05190.5679/0.5513/0.59670.5679 / 0.5513 / 0.5967

Across the sweep, the M-FAC hypergradient stays close to the CG reference at all three points of training, except NL=1N_L=1 at mid-training, where a single minibatch gradient is likely too noisy to estimate curvature well. Increasing NLN_L generally improves the match, and decreasing the LoRA rank rr also tends to improve the match, although the trend is weaker. The cosine similarities of the correction term are overall lower and noisier, but shows the excepted rise with NLN_L, from roughly 0.30.3 at NL=1N_L=1 to 0.6−0.80.6-0.8 for NL≥10N_L\geq10. We have plotted the results in Figure 5 and added the methodology in Section D.2.1.

Q4

4: What is the total cost of the BiSSLight stage plus fine-tuning, compared with PEFT alone?

PEFT alone runs 200 epochs, which is 800-3,800 steps on the datasets used here, costing roughly 0.5−1.50.5-1.5, 1−21-2, and 2−62-6 GPU hours for ViT-B, ViT-L and ViT-H respectively. The BiSSLight stage adds a fixed (NU+NL)T=(8+20)⋅500=14,000(N_U + N_L) T = (8+20)\cdot 500= 14,000 steps on top, taking about 44, 55 and 88 GPU Hours for ViT-B, ViT-L and ViT-H respectively. As shown in our response to Reviewer wmcB (as well as in our response to Q6 below), giving PEFT the same total compute does not close the accuracy gap, and even takes 9-12 GPU hours on ViT-L, exceeding the total runtime of our BiSSLight+PEFT pipeline of 6−76-7 GPU Hours. We will note these compute costs in the paper. (Section D.4.2)

Continued...

AuthorsOpenReview ↗

Response to Reviewer Js5h [3/4]

Q5

5: The Fisher–Hessian equivalence assumes a likelihood-based loss and a well-specified model. The MAE normalized-pixel MSE loss and the I-JEPA latent-prediction loss are not obviously of this form. What justifies the empirical Fisher as a curvature surrogate here?

We appreciate this point being raised, and agree that the theoretical justification for using the empirical Fisher as a Hessian surrogate is less straightforward for self-supervised objectives that lack an explicit likelihood interpretation. Nevertheless, the distinction between supervised and self-supervised objectives does not necessarily invalidate the underlying motivation for using Fisher-based approximations. Fisher-based second-order methods have been applied beyond classification likelihoods, including self-supervised autoencoder training [1]. Moreover, the MSE-based pretext objectives considered here hold a Gaussian negative log-likelihood interpretation, partially providing some theoretical motivation for considering Fisher-based curvature approximations. We nevertheless acknowledge that the extent to which the Hessian–Fisher relationship carries over to these objectives, and the accuracy of the empirical Fisher as a curvature surrogate, is not fully established. We assess its practical accuracy empirically in our response to Q3, while leaving a more thorough theoretical characterization for future work.

[1] James Martens and Roger Grosse, "Optimizing Neural Networks with Kronecker-factored Approximate Curvature", Proceedings of the 32nd International Conference on Machine Learning, 2015

Q6

6: BiSSLight adds roughly 2 ImageNet epochs of MAE training, 4,000 upper-level steps on downstream data, and a 20-epoch linear probe. Can you add compute-matched baselines, such as (a) LoRA continued MAE pre-training with no upper level (similar to ExPLoRA), (b) upper-level/downstream-only training of the same length, and (c) PEFT with an equally extended schedule? This would isolate the benefit of the bilevel coupling itself.

As detailed in our response to Reviewer wmcB, we have added compute-matched baslines of (a) LoRA continued MAE pre-training (ExPLoRA-style, matched to 10,000 (Ext.PTLPT_L + PEFT) or 14,000 steps (Ext.PTL+UPT_{L+U} + PEFT), (b) the same with the subsequent PEFT schedule extended to also cover the upper-level steps (Ext.PTLPT_L + PEFT (CMUCM_U)), and (c) compute-matched plain PEFT trained for a matched total number of steps (PEFT (CML+UCM_{L+U})) to the experimental results. Across all six datasets, these baselines improve only sporadically and sometimes degrade performance relative to PEFT, while BiSSLight + PEFT achieves the best accuracy in the large majority of comparisons. If baseline (b) was intended differently than we understood, we are happy to run it with a clarification. The results are pasted below for clarity and are found in Section 4.4 of the paper.

DatasetPetsDTDVOC07FlowersCUB200Aircrafts
PEFT94.4±0.394.4\pm0.374.1±0.674.1\pm0.685.5±0.285.5\pm0.294.6±0.494.6\pm0.484.4±0.484.4\pm0.479.1±0.479.1\pm0.4
PEFT (CM$_{L+U}$)94.6±0.294.6\pm0.274.0±0.674.0\pm0.685.3±0.385.3\pm0.395.7±0.295.7\pm0.284.3±0.584.3\pm0.581.0±0.381.0\pm0.3
Ext.PT$_L$ + PEFT94.0±0.394.0\pm0.374.0±0.474.0\pm0.484.8±0.284.8\pm0.294.9±0.494.9\pm0.484.7±0.384.7\pm0.381.3±0.381.3\pm0.3
Ext.PT$_L$ + PEFT (CM$_U$)93.9±0.493.9\pm0.473.2±0.673.2\pm0.684.7±0.284.7\pm0.295.4±0.395.4\pm0.384.3±0.484.3\pm0.482.8±0.482.8\pm0.4
Ext.PT$_{L+U}$ + PEFT94.2±0.394.2\pm0.374.0±0.374.0\pm0.384.6±0.384.6\pm0.394.7±0.394.7\pm0.384.1±0.684.1\pm0.681.5±0.681.5\pm0.6
BiSSLight + PEFT94.7±0.294.7\pm0.274.9±0.474.9\pm0.485.9±0.185.9\pm0.195.8±0.295.8\pm0.285.6±0.385.6\pm0.382.2±0.382.2\pm0.3

Continued...

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Overview

We again thank the reviewers for their efforts in reading and reviewing our work. We are pleased that the presentation was found clear and that the efficiency gains and consistent acuracy improvements of our method were acknowledged by all reviewers. We have made concentrated efforts to address all raised concerns, and below we summarize the main shared themes of the reviews and our responses, along with an overview of additional experiments and content included in the revised paper.

Compute-Matched Baselines

Both reviewers Js5H and wmcB asked for compute-matched baselines to assess whether BiSSLight still benefits under a comparable compute budgets. We ran two sets of experiments, (1) subsequent Parameter Efficient Fine-Tuning (PEFT) extended to match the additional compute incurred by BiSSLight and (2) extended MAE pretraining on the downstream data in three variants matching different parts of the total BiSSLight compute. These baselines did not consistently benefit from the extra budget, sometimes underperforming the standard PEFT baseline, while our BiSSLight + PEFT generally achieved notably stronger performance. Included in Section 4.4 and Table 2.

Deeper Assessment of the Hypergradient Approximation

Reviewer Js5H requested a more analytical assessment of the M-FAC hypergradient approximation against a well-converged reference, sweeping key hyperparameters. We conducted this study and included it in Section D.2.1, and found that the M-FAC approximation expectedly improves as more gradients are included in its estimate, and that the deployed hypergradient tracks the reference closely. Reviewer wmcB raised a related question about the advantage of M-FAC over simpler approaches, so we also ran a simpler, yet practically computationally equivalent variant with an equivalent diagonal Fisher approximation. The diagonal Fisher matches M-FAC somewhat closely, but falls slightly below it in general, which we attribute to the diagonal approximation not capturing second-order cross-parameter interactions. Included in Section D.2.2.

Dependence on Pretext Task Access

Reviewer 7Rf6 raised the weakness that the experiments require access to the pretraining dataset and the pretrained pretext head, and Reviewer wmc8 relatedly asked how much BiSSLight depends on pretext access. We added two restricted-access experiments, one using only a 10% subset of the pretraining data and one using a randomly initialized pretext head trained on the frozen backbone. Both match the default BiSSLight + PEFT closely and clearly outperform PEFT, showing the method remains effective under restricted pretext access. Included in Section D.3.

Summary of Revisions

In response to Q1 from Reviewer Js5H we added an asymptotic complexity analysis of BiSSLight versus BiSSL in Section B.1, and we extended the compute and memory benchmark discussion in Section D.4.2 to also cover PEFT runtime.

Overview of added experiments:

  • Compute-Matched Baselines (Extended PEFT + Extended Pretraining), Section 4.4, Table 2.
  • Quantitative Assessment of Attention Rollout Maps (Segmentation Mask Overlap), Section 4.5, Table 3.
  • M-FAC Hypergradient Assessment (Reference + Diagonal Fisher Comparison), Section D.2.
  • Performance Under Restricted Pretext Access, Section D.3.
  • BiSSL and BiSSLight Full Fine-Tuning Runtime/Memory Benchmarking, Section D.4.2, Table 10.

Conclusion

We hope that our rebuttal has addressed the comments received. We believe that the resulting revisions have strengthened the quality of the submission, and that the revised manuscript provides a thorough response to the issues raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.