- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies pseudo-label reliability in continual semi-supervised learning (CSSL), with a particular focus on the non-revisitable nature of the task stream. The proposed Reliability-aware Calibration Framework (RCF) represents each unlabeled sample using two reliability signals. The predictive signal combines prediction confidence with consistency across weakly augmented views, while the semantic signal measures the margin between the predicted class prototype and the strongest competing prototype in a frozen ViT feature space. Then a GMM is fitted to this two-dimensional reliability space. The posterior of the component with the larger predictive mean is treated as a reliability score. The method performs particularly well in the low-label ImageNet-R and ImageNet-A settings, and the paper also includes component ablations.
Critical strengths
- This paper focus on the non-revisitable nature of the task stream. This is meaningful and practical problem and cause a lot of wrong pseudo labels.
- The experiments verify that the method performs particularly well in the low-label ImageNet-R and ImageNet-A settings.
Critical weaknesses
- The theoretical argument for pseudo-label reliability collapse is not fully convincing. Proposition B.3 relies on a gradient-norm comparison that does not generally hold when historical-task and current-task gradients conflict. The current proof seems insufficient to establish reliability collapse as an inherent consequence of the non-revisitable setting.
- Can RCF also improve standard semi-supervised learning when the non-revisitable constraint is removed? Since the proposed predictive-semantic reliability estimator is not obviously specific to continual learning, an experiment on standard SSL would help clarify which gains come from general pseudo-label selection and which are specific to the CSSL setting.
- It remains unclear how general the proposed selector is across different LoRA-based continual learning methods. Could the authors evaluate RCF on another representative LoRA-based method, such as Keep-LoRA, to separate the benefit of reliability-aware pseudo-label selection from that of the underlying continual-learning architecture? In fact, the methods like Keep-LoRA in table 1 are largely worse than RCF. So, can RCF help Keep-LoRA improve performance?
- The experimental scope is still fairly narrow. All experiments are conducted on image classification benchmarks under a class-incremental setting, and the main model always uses the same pretrained ViT-B/16 backbone.
Questions for authors
See in weakness.
7 comments
Response to Reviewer ZH5s (Part 2)
w2. Can RCF improve standard SSL, and which gains are specific to CSSL?
A2. RCF also improves standard SSL. To evaluate its pseudo-labeling mechanism outside continual learning, we conduct joint training on ImageNet-R with 1% labels and seed 1993. All 200 classes are learned simultaneously using a frozen ViT-B/16 backbone and a single LoRA expert, with continuous access to the training pool. This setting removes task sequencing, expert expansion, and task-based routing. The protocol and results are reported in Appendix D.8.
The confidence-based baseline and RCF share the backbone initialization, labeled subset, optimizer, augmentation settings, and number of training updates. Both use two labeled-only warm-up epochs followed by eight semi-supervised epochs, with 150 updates per epoch. Their final checkpoints are evaluated on the same test set, allowing the benefit of RCF's pseudo-labeling pipeline to be assessed within the same single-expert learning framework.
| Method | Test top-1 accuracy |
|---|---|
| Confidence-based baseline | 22.30% |
| RCF | 29.41% |
| Improvement | +7.12 pp |
The improvement is computed from unrounded accuracies. We also evaluate the same trained checkpoints after excluding test images with byte-identical counterparts in the training partition. The resulting 7.09-percentage-point gain confirms that the improvement persists after removing those duplicates.
The standard-SSL result demonstrates that RCF provides a general benefit through its predictive–semantic pseudo-labeling mechanism, without requiring task isolation or expert expansion. The CSSL experiments further demonstrate its effectiveness when training proceeds through sequential task distributions and historical data become inaccessible.
Together, these results highlight RCF's central advantage: it combines predictive consistency with semantic support to guide unlabeled supervision beyond confidence-only selection. Its reliability estimator operates on the available training pool, while adaptive selection accommodates the evolving model. This enables the same framework to improve learning in both joint standard SSL and continual task streams, supporting its use across different training protocols and expert architectures.
Response to Reviewer ZH5s (Part 3)
w3. Can RCF improve Keep-LoRA and other LoRA-based continual learners while keeping their underlying architectures unchanged?
A3. We evaluate RCF as a plug-in pseudo-label selector within Keep-LoRA and CL-LoRA. Each paired comparison retains the underlying architecture, data split, training settings, and evaluation protocol. This allows the benefit of reliability-aware selection to be assessed within the same learning framework, separating it from improvements obtained by changing the continual-learning architecture.
Results on ImageNet-R with 1% labels are summarized below. Values report average incremental accuracy, and improvements are expressed in percentage points.
| Architecture | Confidence-based selector | RCF selector | Improvement |
|---|---|---|---|
| CL-LoRA | 48.10% | 61.44% | +13.34 pp |
| Keep-LoRA | 7.58% | 15.11% | +7.53 pp |
The Keep-LoRA(Keep-LoRA uses feature-based routing without ground-truth task IDs.) comparison directly addresses the reviewer's question: its AIA nearly doubles after integrating RCF, while its underlying architecture is retained. Thus, Keep-LoRA's lower confidence-based performance does not prevent it from benefiting substantially from improved pseudo-label selection. The CL-LoRA comparison(The CL-LoRA selector comparison uses ground-truth task IDs for task-restricted classification.) provides a second within-architecture validation, showing that the benefit extends across both evaluated LoRA-based learners. These transfer evaluations are reported in Appendix D.3.
RCF complements these architectures by improving how unlabeled samples are selected for supervision. It combines prediction consistency with prototype-supported semantic agreement, allowing selection to consider evidence beyond the model's confidence alone. Its distribution modeling and temporal updates then adapt the selection process as the learner evolves.
Together, these results demonstrate that RCF's benefit extends beyond the architecture used in our main experiments. It can strengthen existing LoRA-based continual learners through their pseudo-labeling pipeline while preserving their adaptation mechanisms. This makes reliability-aware selection a transferable component for improving learning under scarce labeled supervision.
Response to Reviewer ZH5s (Part 1)
w1. Can Proposition B.3 establish reliability collapse under non-revisitable learning when task gradients may conflict?
A1. We have revised the motivation and theoretical analysis in both the main text and the appendix. The revised argument establishes how pseudo-label errors affect within-task optimization, with explicit assumptions and bounds.
(1) Correcting the scope of the argument.
The original gradient-norm comparison does not hold in general: opposing historical and current-task gradients can reduce the norm of their sum. It therefore cannot establish reliability collapse as an inherent consequence of non-revisitability.
Our implemented architecture freezes the backbone and previous experts while updating only the active expert. We have accordingly refocused the motivation on within-task supervision: pseudo-labels guide the active expert's updates, and the updated expert generates predictions for subsequent epochs. This correction is reflected in the Abstract, Section 1, Section 3.2, and Appendix B.1.
(2) Quantifying pseudo-label-induced gradient perturbation.
The revised Proposition B.3 compares the pseudo-label objective F(θ) with a true-label reference objective F(θ) on the same selected batch. The reference is used only for analysis. The current update state, selected samples, augmentation views, and unsupervised-loss weight λ are fixed, isolating the effect of label correctness.
For softmax cross-entropy, the difference between the two logit gradients is zero for a correct pseudo-label and has Euclidean norm 2 for an incorrect one. If the logit Jacobians have operator norm bounded by M, then
\qquad \|b(\theta)\|_2\leq\sqrt{2}\lambda M\epsilon,$$ where $\epsilon$ is the selected pseudo-label error rate. This bounds the supervision-induced gradient perturbation linearly in $\epsilon$, without comparing historical and current-task gradients. **(3) Connecting the perturbation to local optimization.** Assume that $F$ is $L$-smooth along the update segment. For the local gradient step $\theta^+=\theta-s\nabla\widehat F(\theta)$ with $0<s\leq1/L$, smoothness gives $$F(\theta^+)\leq F(\theta) -\frac{s}{2}\|\nabla F(\theta)\|_2^2 +\frac{s}{2}\|b(\theta)\|_2^2 \leq F(\theta) -\frac{s}{2}\|\nabla F(\theta)\|_2^2 +s\lambda^2M^2\epsilon^2.$$ The reference-gradient term contributes descent, while pseudo-label errors introduce a quadratic penalty in the bound. Reducing $\epsilon$ tightens both the gradient-perturbation bound and the local optimization bound; when $\epsilon=0$, the penalty vanishes. The revised proposition thus provides a quantitative rationale for accurate pseudo-label selection during active-expert training. It replaces the unsupported non-recoverability claim with an explicit connection between supervision errors and local optimization under the stated conditions.