acceptodds
Under review as a conference paper at ICLR 2027

FluxLoRA: Flux-Limited Dynamical Low-Rank Adaptation under Noisy Pseudo-Labels

Abstract

Vision foundation models (VFMs) provide strong transferable representations for semi-supervised learning (SSL), especially when labeled data are limited. However, fine-tuning VFMs with unlabeled data remains sensitive to pseudo-label quality. Incorrect pseudo-labels can introduce noisy supervision and reduce the benefits of VFM pre-training. To address this problem, we propose FluxLoRA, a VFM fine-tuning method designed specifically for SSL under noisy pseudo-label supervision. FluxLoRA enables more stable and noise-robust adaptation while preserving useful information from unlabeled data. Extensive experiments across diverse SSL benchmarks, label budgets, and frozen VFM backbones show that FluxLoRA consistently improves the performance and robustness of VFM-based SSL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semi-supervised Learning, Image classification

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_jvusOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FluxLoRA, a low-rank adaptation framework designed to mitigate the adverse effects of noisy pseudo-label gradients in semi-supervised vision foundation model adaptation. The paper provides theoretical analyses of diffusion stability, conditional core-gradient denoising, and residual contraction, and evaluates the proposed method on VTAB, CIFAR-10, and CIFAR-100 using CLIP and DINOv3 backbones. Nevertheless, several concerns remain regarding the reliability of pseudo-label-conditioned graph guidance, the attribution of the main performance improvements, and the empirical validation of the proposed geometric mechanisms.

Critical strengths
  1. Well-motivated research problem. The paper identifies an important limitation of conventional pseudo-label filtering and weighting methods: accepted pseudo-label gradients may still introduce harmful updates into low-rank adaptation.
  2. Interesting geometric perspective. The distinction between gradient contamination within a low-rank subspace and residual gradient components that may induce shifts in the subspace itself provides a logically coherent explanation for how noisy pseudo-labels affect low-rank adaptation.
  3. Structured method design. The combination of labeled-anchor-guided diffusion, residual gating, tangent-space projection, and low-rank retraction is technically motivated and reasonably well organized.
  4. Promising empirical performance. The proposed method consistently improves average accuracy across multiple SSL frameworks and two vision foundation models. The gains over standard PEFT baselines on VTAB are substantial.
Critical weaknesses
  1. The paper motivates FluxLoRA by identifying two distinct effects of noisy pseudo-label gradients: within-subspace core contamination and noise-induced subspace drift. Although the mathematical decomposition establishes how these components can affect low-rank adaptation, it does not demonstrate their effects during semi-supervised training. In particular, Figure 1 provides a conceptual illustration rather than direct empirical evidence that incorrect pseudo-labels substantially corrupt core gradients or cause harmful subspace deviations.
  2. The labeled-anchor retrieval mechanism in Eq. (5) selects an anchor according to the teacher-predicted class of each unlabeled sample. When this prediction is incorrect, the retrieved labeled anchor necessarily belongs to the predicted class rather than the sample's true class. Consequently, a correctly labeled anchor may provide semantically mismatched guidance to the unlabeled sample. Although the proposed reliability estimation and flux limiter may attenuate such effects, the paper does not directly examine whether incorrect anchor assignments improve or worsen the resulting gradient directions.
  3. While the ablation studies and matched one-step interventions support the effectiveness of the overall optimization procedure, they do not directly establish that core-gradient correction reduces the discrepancy between pseudo-label core gradients and their clean-label counterparts Similarly, the paper provides a theoretical analysis of subspace drift using principal angles and explicitly introduces principal-angle drift as an evaluation metric in Appendix D.4, but the experimental results do not report quantitative comparisons of subspace geometry across methods.
  4. Limited attribution of the main VTAB performance gains under fully matched settings. The main VTAB results demonstrate substantial improvements over standard LoRA. The existing component ablations and controlled comparisons are valuable and partially address these factors. Nevertheless, the most tightly controlled end-to-end comparisons in Table 3(b) are limited to CIFAR-10 and CIFAR-100 with DINOv3, where fixed-rank FluxLoRA improves over warm-up-matched LoRA by 0.4 and 0.1 percentage points, respectively. It remains unclear how much of the performance gain on the VTAB benchmark can be attributed to the proposed gradient construction method.
  5. Insufficient discussion of recent closely related work. The paper does not adequately position FluxLoRA against recent advances in geometric LoRA optimization and noise-robust adaptation. In particular, LoRA meets Riemannion (ICLR 2026) and Riemannian Optimization for LoRA on the Stiefel Manifold (EMNLP 2025) are highly relevant to its manifold-based optimization framework. Recent gradient-aware adaptation methods, including RaLoRA and IGU-LoRA (ICLR 2026), as well as reliability-aware semi-supervised adaptation methods such as TriNoL (2026 preprint), are also not discussed. These omissions make it difficult to assess the methodological novelty of FluxLoRA relative to recent developments.
Questions for authors
  1. An analysis of standard LoRA under noisy pseudo-labels, comparing core-gradient errors and subspace evolution between correctly and incorrectly pseudo-labeled samples, would help establish the practical significance of the proposed core contamination and subspace drift mechanisms.
  2. An analysis separating correctly and incorrectly pseudo-labeled samples, together with comparisons of alternative anchor retrieval or graph filtering strategies, would help establish when the proposed graph guidance is reliable.
  3. A direct comparison of core-gradient alignment or estimation error before and after diffusion, particularly for incorrectly pseudo-labeled samples, would help establish the effectiveness of Core Correction. Similarly, comparing principal-angle trajectories or projection-matrix distances between FluxLoRA and relevant baselines would provide stronger evidence for the effectiveness of Residual Gating in mitigating noise-induced subspace drift.
  4. The results in Table 3(b) do not invalidate the larger VTAB gains, since the experiments involve different datasets and settings. However, comparable initialization-, rank-, and optimizer-matched evaluations on the VTAB benchmarks would clarify how much of the headline improvement can be attributed specifically to the proposed gradient construction. Statistical uncertainty should also be reported for the smaller controlled gains.
  5. The authors should discuss the closely related studies mentioned in the Weakness (5) to verify the novelty of FluxLoRA relative to recent developments.
Reviewer_7c8JOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces FluxLoRA, an optimization-level variant of LoRA tailored for adapting vision foundation models in semi-supervised settings with noisy pseudo-labels. By leveraging an orthogonal factorization ΔW=USV⊤\Delta W = U S V^\top, the method decomposes each per-sample pseudo-label gradient into a core component U⊤G~iVU^\top \tilde{G}_i V and an out-of-core residual. To handle noise, it refines the core components via flux-limited nonlinear diffusion over a batch-level feature graph—using labeled gradients as fixed boundary states—while suppressing the residuals through a reliability gate. The resulting aggregated update is then projected onto the fixed-rank tangent space and updated via truncated-SVD retraction (with an option for adaptive rank allocation). Empirical evaluations across six VTAB tasks (under two label budgets) and CIFAR-10/100 using CLIP and DINOv3 ViT-B/16 backbones show promising results, outperforming the strongest baselines by an average of 6.1 and 4.8 points on VTAB, respectively.

Critical strengths
  1. The core/residual decomposition and its tangent-space interpretation (Eq. 15, Thm. A.8) are clean and easy to follow.
  2. The method is complementary to pseudo-label selection and refinement methods and is evaluated with several base SSL algorithms.
  3. The numerical well-posedness of the diffusion solver (Lemmas A.1, A.4) is analyzed carefully.
Critical weaknesses

Weaknesses

The gains seem too good for what the method does. At the default rank, the object being diffused is a tiny matrix per layer on a mini-batch graph. Yet the paper reports improvements of several points on VTAB for every base SSL method. I would need much more convincing evidence of where this comes from. The ablations make this harder, not easier: the variant with no diffusion at all still clearly beats the LoRA baselines, and removing the labeled anchors is worse than removing diffusion entirely. Neither result is discussed.

The comparisons are not clean. Besides the gradient construction, FluxLoRA also changes the warm-up, the optimizer (plain Riemannian steps instead of AdamW after warm-up), and the rank schedule. When the paper does control for warm-up and schedule (Table 3b), the advantage over LoRA almost disappears. I also could not tell whether the main VTAB baselines received the same warm-up.

The theory doesn't add much. Theorem A.8 is essentially a block decomposition of a matrix. The denoising result relies on an assumption that the authors themselves say does not hold for the algorithm they actually run, and the drift bound has unspecified constants. None of it says anything about accuracy, and the many disclaimers throughout the paper more or less admit this.

Some evidence is thin or missing. The one-step direction study shows very small differences with no number of trials or error bars. The appendix promises memory and inference-time measurements that I couldn't find. AdaLoRA and DoRA appear in Table 9 without citations or any description of the setup.

Internal Inconsistencies in the Results

This is my main concern. Several things in the tables don't add up:

  • In Table 10, the standard deviations for every FluxLoRA row are exactly the same under CLIP and DINOv3, setting by setting. This does not happen for the baselines, and I can't see how it could happen with independent runs.
  • In the sensitivity tables (Tables 5–7), almost every DINOv3 number is the CLIP number plus the same constant, no matter which hyperparameter is changed.
  • Table 1 reports FluxLoRA with ST and PET, but those rows are missing from the per-setting table it refers to.
  • The text says FluxLoRA improves every SSL method on both backbones, but Table 1 has at least one case where it is slightly worse (DINOv3 + FineSSL on CIFAR-10). The "strongest baseline" margin in the abstract also ignores DoRA from Table 9. I would like the authors to explain these and share their per-seed logs. Until then, I find it hard to take the empirical claims at face value.

Presentation

Honestly, the paper is hard to read. Most figures are too small to make out without heavy zooming, especially the legends and the annotations in Figure 1. Figure 2 mixes two unrelated visual styles, and the caption of Figure 3 runs straight into the caption of Table 2. The tables in the appendix all look different from each other, and Table 10 is barely legible.

Algorithm 1 reads more like a checklist of implementation details than pseudocode, and the main idea gets lost in it. There are also broken references ("Eq. equation 104"). The text repeats itself a lot: the same remarks about step sizes and node degrees show up in both the method section and the appendix, there are duplicated sentences in Section 3.2, and there is a stray paragraph at the end of Appendix C that doesn't connect to anything around it.

Questions for authors
  1. Can you explain the identical standard deviations in Table 10 and the constant backbone offset in the sensitivity tables, and share per-seed results?
  2. Where are the per-setting results for FluxLoRA with ST and PET?
  3. Did the LoRA and AdaptFormer baselines use the same labeled-only warm-up? Which base SSL method is used in Table 3(b)?
  4. Where does the gain of the no-diffusion variant over LoRA come from, and why is removing the anchors worse than removing diffusion altogether?
Reviewer_UFfFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new SSL method to train effective ViT models using Lora in SSL setups. Specifically, it tackles the bad influence of incorrect pseudo-labels during training but instead of only utilizing a weighting mechanism like other approaches (e.g., FlexMatch, SoftMatch), it proposes to also split each pseudo-label gradient into an r x r core and an outside-core residual. The core controls updates within the current input and output subspaces of LoRA and is corrected through diffusion using labeled anchors. The residual contains directions that rotate these subspaces and is scaled by a reliability gate to limit the influence of unreliable samples. During update, in order to clean up noisy gradients, the method builds a similarity graph of the batch. It then connects attached unlabeled samples with the same predicted class to a clean labeled anchor sample from the same class, then runs diffusion so gradients are pulled towards gradients of clean samples. Additionally, similar to other SSL approaches, the method continues to utilize a reliability gate based on confidence and augmentation consistency.

Critical strengths
  • Overall the paper is well written and easy to understand.
  • The idea is well-motivated and an interesting new way of thinking about training under label noise: splitting lora into “updates with the current subspace” and “updates that rotate the subspace”.
  • Reported improvements are broad across both backbones, all eight SSL methods, and nearly all settings.
Critical weaknesses
  • It’s still unclear where the gains come from. FluxLoRA changes initialization, optimization, and adaptation rank alongside its gradient processing. Table 3 provides useful controlled CIFAR comparisons, but are missing from VTAB results. Would be good to compare against LoRA and a raw-gradient manifold baseline with matched rank, warm-up, and tuning.

  • For every batch of 32 unlabeled images, FluxLoRA fetches up to 32 extra labeled anchor images and computes gradients. This means FluxLoRA draws gradient signals from up to 64 labeled samples per step while standard LoRA only sees 32. This may explain why "No labeled anchors" causes the biggest accuracy drop in the ablations (-4.8 to -5.3 points in Table 2)? Would be good to test standard LoRA with those same extra labeled samples.

  • The fixed-rank improvements in table 3(b) are small and lack uncertainty estimates and table 3(a) does not show numer of trials. Would be good to show paired seed-level results and confidence intervals, as well as GPU memory and runtime breakdown of anchor processing, teacher inference, diffusion, retraction. This could show if the additional complexity provides enough benefit.

Questions for authors
  • How does standard LoRA (rank 16 + warm-up) perform on VTAB and CIFAR when it is also given the extra retrieved labeled anchors in its training batch?

  • Also, why does FluxLoRA “Labeled only” beat LoRA “Labeled only” in Table 1 when there are no unlabeled samples?

  • Since an unlabeled sample with a wrong pseudo-label is attached to a labeled anchor from that same wrong class, how can diffusion move its gradient toward the true class?

  • D.4 Appendix mentions that the paper reports anchor alignment split by correct vs. wrong pseudo-labels, principal-angle subspace drift, and peak GPU memory. However, these are not present in the paper.

Reviewer_ATqfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes FluxLoRA, a parameter-efficient fine-tuning (PEFT) method for semi-supervised learning (SSL) with vision foundation models. The authors distinguish two effects of noisy pseudo-label gradients: core contamination and subspace drift. They propose separate mechanisms to address these issues, combining graph-based gradient diffusion, reliability gating, and low-rank manifold optimization. Experiments on VTAB and CIFAR datasets demonstrate improvements over several baselines.

Critical strengths
  1. Promising empirical results. The proposed method achieves strong performance on VTAB and CIFAR datasets, with consistent improvements across different foundation models and SSL methods.
  2. Extensive experiments. The paper includes a broad range of experimental comparisons and ablation studies to evaluate the proposed components.
Critical weaknesses
  1. Poor presentation. The paper suffers from numerous presentation issues, including unnecessarily heavy notation, difficult-to-follow equations, small figures, and inconsistent visual styles. The writing is also excessively verbose, making the paper unnecessarily difficult to read and obscuring its central contributions. Overall, the presentation substantially undermines the clarity and accessibility of the work.
  2. Excessive methodological complexity. The proposed method appears significantly overengineered relative to the problem it aims to address. Starting from the relatively straightforward observation that unreliable pseudo-labels can lead to harmful LoRA updates, the authors introduce a complicated pipeline involving gradient decomposition, graph construction, anchor retrieval, edge weighting, nonlinear diffusion, fidelity control, residual gating, manifold optimization, and optional adaptive rank. While these components are individually motivated, their combination feels more like an accumulation of techniques than a coherent and necessary solution. The practical value of such a complicated framework remains unclear, especially given the relatively modest gains in the controlled comparisons in Table 3(b).
Questions for authors

No specific questions at this stage. My main concerns are the excessive methodological complexity and poor presentation, which make it difficult to identify and evaluate the paper's central contribution. The writing is also verbose and often reads like AI-generated academic prose rather than a carefully developed scientific argument. Trying to understand this paper is torture.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.