acceptodds
Withdrawn from ICLR 2027

Recovering Silenced Gradients: Noise-Robust Prompt Tuning for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prompt tuning efficiently adapts vision-language models by learning a small set of parameters while keeping the backbone frozen. However, existing noise-robust methods suppress useful learning signals along with corrupted supervision. Through theoretical and experimental analyses, we identify a failure mode termed gradient silencing. Correctly labeled samples with low prediction confidence, which we call hard positives, can be misclassified as noisy. Mean absolute error (MAE) then attenuates their corrective gradients in proportion to that confidence. To address this limitation, we propose Geometric Gradient Rectification Prompting (GGRP), which couples geometric target rectification with silenced gradient recovery. The key idea is to use image–text similarities to determine both training targets and sample contributions, rather than only separating clean and noisy samples. Specifically, temporal momentum guidance stabilizes textual class prototypes. We then use geometric target rectification to construct training targets via optimal transport (OT) alignment. Finally, silenced gradient recovery uses the OT assignment margin to balance cross-entropy on rectified targets with MAE on observed labels. This allows hard positives to receive corrective gradients even at low prediction confidence when OT sufficiently supports their correct labels, while retaining MAE-based attenuation for ambiguous samples. Extensive experiments on seven datasets with synthetic label noise and Food101N with naturally noisy labels show that GGRP surpasses state-of-the-art methods across a range of noise settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Prompt Tuning, Vision-Language Models, Learning with Noisy Labels, Robustness, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dTiFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geometric Gradient Rectification Prompting (GGRP) for noise-robust adaptation of vision-language models. GGRP combines EMA-based textual prototypes, optimal transport over a batch-adaptive class support to generate rectified targets, and an assignment-margin gate that balances cross-entropy on rectified targets with MAE on observed labels. Experiments evaluate the method on multiple datasets with synthetic label noise and on Food101N with naturally noisy labels.

Critical strengths

1.The paper is generally well organized, with a clear progression from problem analysis to methodology and empirical evaluation.

2.The experimental evaluation is reasonably extensive, covering multiple datasets, noise types, and corruption rates, together with component ablations and hyperparameter sensitivity analyses.

Critical weaknesses

1.The methodological novelty is questionable. NLPrompt already uses optimal transport to partition examples and applies CE and MAE to different subsets. The performance gains over the strongest competing method are also generally modest, particularly compared with the more substantial improvements emphasized over NLPrompt.

2.The roles and necessity of individual components is not sufficient. The “Silenced Gradient Recovery” subsection introduces the margin-based gate, while the “Hybrid Optimization Objective” subsection incorporates that gate into the CE/MAE objective. These are closely related parts of the same mechanism, but their presentation is somewhat repetitive and does not adequately explain why this particular combination is preferable to simpler alternatives.

Questions for authors

Other weaknesses: 1.The connection between the theoretical analysis and the proposed method remains incomplete. The analysis correctly shows that MAE and GCE attenuate gradients when the target-class probability is low. However, this established property does not directly justify the specific support-selection, target-rectification, and thresholding mechanisms introduced by GGRP. The recovery analysis is conditional on OT providing a correct target with a sufficiently large gate, leaving the reliability of this condition largely unresolved.

2.Some notation and diagnostic definitions are ambiguous. The observed label is defined as tilde y in line 238, while the generic class index is expressed as (c) in eq.7. More importantly, Equation (5) defines rejection through membership in a noisy subset, whereas GGRP uses continuous weighting. The paper should clarify how rejection is defined for GGRP in Figure 1 and whether this measure is directly comparable across methods.

3.Hyperparameter robustness and practical selection remain insufficiently demonstrated. The method introduces several hyperparameters, while the reported sensitivity studies examine individual choices on single datasets rather than establishing that a common configuration transfers reliably across datasets and noise conditions.

4.The related work discussion is too limited to establish the paper’s position. A more thorough discussion should cover classical sample-weighting and robust-loss approaches, optimal-transport-based noisy-label learning methods, and robust prompt tuning for VLMs.

Other questions: 1.Some CoOp results appear to coincide with those reported in the NA-MVP paper, whereas other baseline results differ. Could the authors clarify which results are reproduced and which are taken from prior publications? If the evaluation protocols differ, will this introduce unfairness?

2.How are the gating thresholds and other hyperparameters selected?

Reviewer_F781OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geometric Gradient Rectification Prompting (GGRP) for adapting VLMs (e.g., CLIP) under noisy label. The authors identify gradient silencing phenomenon, where correctly labeled but difficult samples are treated as noisy and receive weak gradients under conventional MAE loss. GGRP maintains an EMA prompt to stabilize textual prototypes and constructs rectified targets through optimal transport over batch-adaptive candidate classes. An OT assignment margin then controls the mixture of cross-entropy on rectified targets and MAE on observed labels. Experiments on seven datasets with synthetic label noise and Food101N demonstrate improvements over NLPrompt, particularly under severe corruption, and competitive results against NA-MVP.

Critical strengths
  1. The paper addresses a meaningful limitation of noise-robust prompt tuning: suppressing mislabeled samples can also weaken learning from correctly labeled hard samples. The motivation is clearly presented.
  2. The method is reasonable and straightforward. EMA guidance, active-support OT, and margin-based loss weighting provide complementary mechanisms for stabilizing target construction and controlling potentially harmful updates.
  3. The overall manuscript is well-written and easy to follow.
Critical weaknesses
  1. The paper provides insufficient evidence that the proposed method effectively restores corrective gradients for hard positive samples. The theoretical analysis in Appendix B is conditional on correct OT assignments and does not directly establish how often such recovery occurs in practice. Direct empirical evidence, such as visualizations quantifying the proportion of hard positive samples that are correctly assigned and receive restored corrective gradients, is also missing.
  2. The conceptual novelty appears somewhat incremental. The difficulty of learning hard samples with MAE and its gradient-weighting relationship with CE were already analyzed in GCE (NeurIPS 2018), while NLPrompt (CVPR 2025) already combines OT-based sample partitioning with CE/MAE training. The paper should more clearly distinguish its new insight from these established observations and isolate the contribution of target rectification from EMA guidance and continuous loss weighting.
  3. The performance improvement over the strongest compared method is modest and inconsistent. Evaluations on larger datasets (ImageNet and SUN397) should be included. The experiments on other backbone such as CLIP-Vit/B-16 are also important.
  4. The related work is insufficient. More recent and advancing methods of prompt learning and noisy label learning are suggested to be added.
Questions for authors

Please refer to the weaknesses.

Reviewer_43pZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prompt tuning of CLIP under noisy labels. The authors argue that NLPrompt, which partitions training samples into clean and noisy subsets according to whether the labels assigned by optimal transport (OT) agree with the observed labels, and which applies cross-entropy (CE) to the clean subset and mean absolute error (MAE) to the noisy subset, can incorrectly assign correctly labeled samples on which the model has low confidence (hard positives) to the noisy subset. Because the MAE gradient is scaled by the predicted probability of the labeled class, these samples receive nearly vanishing corrective gradients, a phenomenon the authors term gradient silencing.

To address this limitation, the authors propose GGRP. Like CoOp, GGRP trains only the soft prompt; it changes the per-sample training targets and the choice of loss rather than the set of learnable parameters. The method consists of three components:

EMA teacher: An exponential moving average of the prompt parameters (θ′) is maintained to construct more stable text class prototypes for OT.

Batch-wise OT pseudo-labeling: Within each mini-batch, samples are assigned to a set of candidate classes, defined as the union of the observed labels in the batch and the top-r (r = 3 by default) classes predicted for each sample. Class-wise capacity constraints derived from the empirical (noisy) label frequencies are imposed, yielding OT-based pseudo-labels.

Margin-based gating: The margin δ between the highest and second-highest OT assignment scores determines the training objective. Samples with large margins are trained with CE on the OT-based pseudo-labels, samples with small margins are trained with MAE on their observed labels, and the two objectives are linearly interpolated for intermediate margins.

The method is evaluated with a CLIP ResNet-50 backbone in a 16-shot setting on seven datasets under symmetric and asymmetric label noise (12.5% to 75%), as well as on Food101N with real-world label noise. The authors report improvements over NLPrompt that become more pronounced at higher noise rates.

Critical strengths
  1. The problem diagnosis is specific, and the proposed solution is closely aligned with the identified issue.
  • The authors connect the probability-dependent gradient scaling of MAE and GCE (proportional to s_ỹ, or s_ỹ^q for GCE) to the failure mode of hard sample partitioning.
  • Quantify the phenomenon with R_hard and η_t, providing empirical evidence of gradient silencing in NLPrompt (Fig. 1).
  • The logit-gradient analysis in Appendix B.2 supports the proposed mechanism: when w = 1, the gradient with respect to the target logit approaches −1 as the target probability goes to zero, whereas when w = 0 the objective reduces to MAE.
  1. The paper provides extensive benchmark evaluations, well-designed controlled experiments, and a relatively transparent discussion of limitations.
  • The evaluation covers seven datasets, two noise types, and six noise rates, together with Food101N, and standard deviations are reported in the appendix.
  • The controlled variants in Table 4 are particularly informative. On Caltech101 with 75% symmetric noise, replacing the OT-based targets with the EMA teacher's argmax predictions while keeping the same gate reduces accuracy from 87.86% to 77.86%, which indicates the importance of the class-capacity constraints in OT. Applying CE to all OT targets without the gate reduces accuracy to 69.57%, which highlights the role of the proposed gate.
  • The authors acknowledge the mixed results relative to NA-MVP and discuss limitations related to support coverage, which makes the presentation more balanced.
Critical weaknesses
  1. The empirical evidence for the claimed "silenced gradient recovery" is insufficient.
  • The contribution of the CE branch is evaluated only on Caltech101, where performance is close to saturation. This makes it difficult to determine whether the mechanism recovers informative gradients in more challenging settings, where gradient silencing should have a greater impact.
  • It remains unclear whether hard positives actually receive CE-based supervision, and the gating rule may introduce new cases of gradient silencing. The gating weight w depends only on the OT margin δ, not on whether the OT pseudo-label agrees with the observed label. As a result, a sample whose OT pseudo-label agrees with its observed label but whose margin is below τ_low is trained with MAE only. NLPrompt's agreement-based partition, in contrast, would treat a sample with an agreeing OT label as clean and train it with CE. Some correctly labeled hard positives may therefore receive weaker corrective gradients under GGRP than under NLPrompt.
  • The novelty of the gradient analysis is somewhat limited. The probability-dependent gradient scaling of MAE and GCE was already discussed in the original GCE paper (Zhang & Sabuncu, 2018). The connection to hard sample partitioning is relevant, but the paper would benefit from distinguishing more clearly what is new in its analysis relative to existing results.
  1. The claim that GGRP's advantage grows with the noise rate holds relative to NLPrompt but is not consistently supported relative to the strongest baseline, and some experimental conditions require clarification.
  • The improvement over NA-MVP is limited, and GGRP underperforms NA-MVP in several high-noise settings. On UCF101 with asymmetric noise at 50%, 62.5%, and 75%, GGRP is lower by 0.98, 2.58, and 3.69 percentage points, respectively. On StanfordCars with asymmetric noise at 62.5% and 75%, it is lower by 1.38 and 3.41 points. In these settings, the gap widens as the noise rate increases. The paper states that its gains become clearer under severe noise (relative to NLPrompt; L431, L720), but this trend does not extend to NA-MVP on these datasets, and the per-dataset results are not discussed.
  • The comparison conditions and the batch construction procedure are insufficiently specified. Since OT is solved at the batch level, batch composition directly affects the rectified targets, yet the class-aware sampling strategy is described only briefly in Algorithm 1 and Appendix C. More details are needed to assess the reproducibility and robustness of the method.
Questions for authors
  1. Ablations in more challenging settings. On Caltech101, the competitive methods all reach roughly 87–92% accuracy at 25–75% symmetric noise (Table 1h), and the stronger variants in Table 4 differ only slightly, which makes it difficult to assess the contribution of each component. Could the authors report the same ablation, including Active OT + MAE versus full GGRP and the other variants in Table 4, in settings where GGRP shows larger gains? For example, under 75% symmetric noise GGRP outperforms NLPrompt by 12.13 percentage points on OxfordPets and by 4.65 points on DTD. Means and standard deviations over multiple seeds would be particularly helpful.
  2. Gradient recovery for hard positives. Among hard positives (the bottom 20% of clean samples by confidence, as defined in Section 3.2), what proportion receives a positive gating weight (w > 0) together with a correct rectified target? Could the authors also report the accuracy of the rectified targets separately for truly clean and truly mislabeled samples, stratified by gating region (w = 0, 0 < w < 1, and w = 1)? In particular, what proportion of hard positives has y^ot = ỹ but δ < τ_low and therefore receives only MAE supervision, although an agreeing OT label would lead to CE training under NLPrompt?
  3. Novelty of the gradient analysis. Could the authors clarify which aspects of the analysis in Section 3 go beyond the gradient analysis in Zhang & Sabuncu (2018)?
  4. Statistical significance and claims of superiority. Many of the average differences between GGRP and NA-MVP appear to be within run-to-run variability. Could the authors provide significance tests or results over additional random seeds? Given the mixed results relative to NA-MVP, would the authors consider qualifying the statement that GGRP "surpasses state-of-the-art methods" in the abstract to better reflect the evidence?
  5. Performance under severe asymmetric noise. On UCF101 with asymmetric noise at 50%, 62.5%, and 75%, GGRP underperforms NA-MVP by 0.98, 2.58, and 3.69 percentage points, respectively, and a similar trend appears on StanfordCars at 62.5% and 75%. Since the gap widens as the noise rate increases, could the authors explain these results in light of the paper's emphasis on larger gains under more severe label noise?
  6. Fairness and reproducibility of baseline comparisons. Were the results for CoOp, JoAPR, NLPrompt, and NA-MVP obtained by rerunning these methods under the same experimental settings, or taken from the original papers? How many random seeds were used for the baseline results, and how many for GGRP?
  7. Details and sensitivity of batch construction. Since batch-wise OT directly determines the rectified training targets, the batch sampling procedure is an important part of the method. Could the authors describe the class-aware sampling strategy in more detail, including the number of classes per batch, the number of samples per class, the definition of "recent" batches, and how an epoch is defined? Are there results on sensitivity to the batch size B?

Clarifications

  1. Heading capitalization. Several paragraph headings in Section 4.2 begin with lowercase letters ("temporal momentum guidance," "geometric target rectification," and "silenced gradient recovery"). Their capitalization should be made consistent with the rest of the manuscript.
  2. Redundant explanations. The discussion of GCE in Section 3.1 substantially overlaps with Appendix B.1. The authors may consider consolidating the two to improve conciseness.
  3. Limited related work. The related work section consists of only two paragraphs and does not sufficiently cover several closely related approaches. In particular, JoAPR and NA-MVP, which serve as experimental baselines, are not discussed there. The paper would also benefit from a more explicit comparison with prior work on OT-based pseudo-labeling and label correction, clarifying how GGRP differs from these approaches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.