acceptodds
Withdrawn from ICLR 2027

Beyond Pseudo-Labeling: Retrieval-Augmented Visual Tuning for Semi-Supervised Learning in the Wild

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Semi-supervised learning (SSL) commonly exploits unlabeled data via confidence-thresholded pseudo-labeling, assuming that high confidence implies correctness. This assumption often fails in in-the-wild regimes with strong distribution mismatch, where thresholding induces two-sided selection bias (admitting high-confidence errors while discarding informative low-confidence samples) and shift further aggravates confidence miscalibration. Their coupling makes single-threshold heuristics brittle, leading to confirmation bias and even the "unlabeled hurts” phenomenon. We propose Retrieval-Augmented Visual Tuning (RAVT), which replaces pointwise pseudo-label acceptance with evidence-backed neighborhoods conditioned on labeled queries. RAVT constructs reliable neighborhoods through hybrid retrieval that combines global semantic recall with patch-level ranking, and strengthens semantic consistency using implicit textual cues; learning is driven by non-local neighborhood evidence and followed by supervised anchored tuning to stabilize representations. Across four progressively harder mismatch settings, RAVT is particularly effective in in-the-wild regimes, including cross-dataset and multi-source heterogeneous unlabeled pools, where it consistently mitigates negative transfer and delivers more stable cross-distribution generalization, achieving substantial gains over strong pseudo-labeling baselines; in standard near-i.i.d. SSL settings, RAVT can also serve as a plug-in component for existing pseudo-labeling methods and further improve their performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semi-supervised learning, Retrieval-Augmented Visual Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1brkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAVT for semi-supervised classification with mismatched unlabeled data. Instead of confidence-based pseudo-label selection, labeled images retrieve neighborhoods through global visual similarity and patch-level ranking. Visual and BLIP-2-derived features are fused, neighborhood predictions are trained against the labeled query, and supervised tuning subsequently anchors the classifier. Evaluation covers in-distribution, transductive, cross-dataset, and multi-source settings, with additional domain-shift and component analyses. Results show substantial improvements under strong mismatch and complementary benefits when combined with conventional pseudo-labeling.

Critical strengths

The paper addresses a consequential limitation of semi-supervised learning: unlabeled data can hurt performance under distribution mismatch. RAVT achieves clear gains in the cross-dataset and multi-source settings. For example, it reaches 90.19% on CIFAR-10 in Setting III, compared with 82.14% for V-PET. These results support its practical value when unlabeled data comes from uncontrolled sources. The evaluation covers several forms of mismatch, including cross-domain shifts, heterogeneous data sources, and class imbalance. RAVT also improves existing SSL methods when combined with them. This breadth strengthens the evidence that its usefulness extends beyond a single dataset or training recipe.

Critical weaknesses

Neighborhood averaging does not explicitly protect against incorrect neighbors. Equation (2) averages all neighbor logits before applying cross-entropy. Each neighbor therefore receives the same gradient with respect to its logits, including off-class neighbors. The statement that these neighbors “only affect the aggregated evidence” does not explain why their harmful supervision is reduced. For a linear classifier, this operation also reduces exactly to classification of the mean fused feature.

Stage II leaves the source of the final gains unclear. The visual encoder remains frozen, and Stage II trains only the classifier on labeled visual features. The textual adapter and retrieved evidence are absent from this objective. Stage I therefore influences the final predictor through classifier initialization. The paper does not establish whether this benefit survives sufficiently converged supervised training or depends on the stopping schedule.

The error decomposition does not establish the claimed selection-bias mechanism. Section 3.1 rewrites selected pseudo-label error using confidence and calibration error. Its first term measures residual uncertainty within the selected set. It does not measure rejected correct samples or changes in class coverage. The identity therefore does not establish the proposed two-sided selection bias or explain why retrieval resolves it.

The leakage controls do not isolate the claimed cause of improvement. Appendix B tests alternative pseudo-label sources and replaces BLIP-2 with CLIP. These experiments show robustness to those choices, but they do not rule out dependence on pretrained semantic knowledge. In particular, poor performance from label-informed pseudo-labels does not establish that equivalent knowledge cannot benefit RAVT through feature fusion and retrieval.

Questions for authors

What evidence shows that averaging logits suppresses off-class supervision, rather than merely reducing prediction variance? How does RAVT compare with classification of mean neighbor features and with loss averaging over individual neighbors?

Does the Stage I advantage persist when Stage II is trained to convergence under the same regularization? How sensitive is the gain to Stage II duration?

Can the authors quantify rejected-correct coverage and class-selection distortion separately from calibration error, then show which quantities retrieval improves?

What comparison isolates retrieval from pretrained semantic transfer while matching the encoders, fused features, trainable parameters, and training schedule?

Reviewer_XngKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies semi-supervised image classification when the unlabeled pool comes from a different distribution than the labeled data. It argues that confidence-thresholded pseudo-labeling fails in this regime because selection bias and shift-induced miscalibration are coupled. The proposed RAVT uses each labeled image as a query to retrieve KK unlabeled neighbors with a hybrid retriever that combines global-embedding k-NN and PatchLex. Each neighbor’s frozen visual embedding is averaged with a linearly projected BLIP-2 feature, and classifier logits averaged over the neighborhood are trained against the query’s label (Stage I); the classifier is then fine-tuned on labeled data alone (Stage II), and retrieval is disabled at inference. On CIFAR-10/100 with 4 labels per class, ViT and ALBEF backbones, and four protocols of increasing mismatch, RAVT outperforms pseudo-labeling baselines in the cross-dataset and multi-source settings, while baselines stay competitive in near-i.i.d. settings, where the authors report that combining RAVT with them gives the strongest overall performance.

Critical strengths
  1. The paper targets a practically important regime, SSL with cross-dataset and multi-source unlabeled pools. Its four-level protocol exposes “unlabeled hurts” in strong baselines.
  2. The design is simple and cheap to adopt: only a linear projection and a classifier are trained on frozen encoders, and retrieval is off at inference. With product quantization and asynchronous updates, training cost is close to the baselines.
  3. Ablations cover every component, and PatchLex alone beats global retrieval. The authors also address the BLIP-2 confound with BLIP-2-equipped baselines, a worst-case leakage control, and CLIP/DINO-v3 swaps.
Critical weaknesses
  1. The gains are not clearly attributable to neighborhood evidence. Without BLIP-2, RAVT falls below the strongest baseline in three of four mismatch cells, and the BLIP-2 feature alone nearly matches the full model. A labeled-only classifier on the same fused features is missing, so the central claim of Sec 4.3 is not established.

  2. The error decomposition in Sec 3.1 is an algebraic identity whose terms are never measured. RAVT’s own supervision error is not analyzed either: under Eq. (2), every neighbor receives the gradient 1K(softmax(hˉ(q))−eyq)\frac{1}{K}(\mathrm{softmax}(\bar h(q))-\mathbf{e}_{y_q}), so off-class neighbors are still pushed toward the query’s label, and no neighbor-purity statistics are reported.

  3. No seeds or variance are reported despite only 4 labels per class, and key training details are missing: epochs, batch size, and whether baselines fine-tune the backbone or train only a head like RAVT. Hyperparameters are described as cross-validated, yet App. C.2 adopts K=26K=26 because it has the highest average over four cases whose values coincide with the Table 1 test results.

  4. The paper names shift-aggravated miscalibration as one of the two causes of pseudo-labeling failure. But in this paper, no experiment measures calibration for RAVT or the baselines in Settings I–IV, which weakens both the motivation and the persuasiveness of the paper.

  5. The paper offers no theoretical analysis of RAVT. Its only formal content is the decomposition, an algebraic identity about pseudo-label error, and no result shows why neighborhood aggregation should be more robust than pointwise selection under shift.

Questions for authors
  1. Do the baselines train the same parameters as RAVT, and how many runs underlie each number? What data were used for cross-validation with only 4 labels per class, and how is “RAVT + X” implemented?
  2. How does RAVT compare with OpenMatch, IOMatch or Safe-Student in Settings III/IV, both alone and combined with RAVT?
Reviewer_g3MUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework to tackle the pseudo-labeling problem in the context of semi-supervised image classification. The authors start with the observation that pseudo-labeling using a fixed threshold fails, especially under distribution shift. The authors then go on to propose a pseudo-labeling method that is more robust, using neighborhood heuristics as well as both local and global visual features extracted from a pretrained model.

Critical strengths
  • The experiments seem expensive.
  • The method seems to have some complexity, as exemplified by Figure 3.
Critical weaknesses
  • Although the method appears complex, I would argue that it is more of an Engineering pipeline consisting of well-known components. Specifically:
    • Using neighborhoods for ranking goes back to CVPR 2017 "Re-ranking Person Re-identification with kk-Reciprocal Encoding". The authors use a similar idea for pseudo-labeling.
    • Using both local and global embeddings for ranking can be attributed to multiple works, landmark recognition papers in particular. In the era of ViTs, this is an especially common trick.
    • The fusion tricks presented are well known to the industry.
  • Some of the method is unclear. For example, in "Stage I", authors use imprecise phrases like "aggregate predictions across the non-local neighborhood". I think there needs to be a precise mathematical procedure.
  • I notice the authors are using old datasets such as CIFAR, OfficeHome, and PACS. I think these datasets are outdated and have largely been solved. This limits the impact of the results.
Questions for authors

I don't agree with using a FAISS index. The datasets here seem small, why not just use exhaustive search?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.