acceptodds
Under review as a conference paper at ICLR 2027

Progressive Trust Calibration for Noisy Continual Learning

Abstract

Learning under noisy supervision in continual environments presents a fundamental challenge that existing methods are ill-equipped to address. Label corruption and distribution shift co-evolve across tasks, so that one-shot static noise-removal rules cannot keep pace with the evolving stream. In this work, we revisit this problem through a fresh lens and argue that sample reliability should not be treated as a fixed binary attribute, but rather as a dynamic property that evolves alongside the model's growing semantic competence. Motivated by this insight, we propose Progressive Trust Calibration (PTC), a unified framework that reframes noisy-label continual learning as a process of sample trust evolution. PTC operates through three synergistic stages. First, Relational Trust Initialization anchors reliability estimation in the intrinsic geometry of a frozen pretrained representation space, measuring neighborhood label consistency and local feature compactness to produce stable, prediction-independent trust scores that are immune to early-stage classifier bias. Second, Prototype-guided Trust Refinement elevates this local assessment to a global semantic level by constructing class prototypes from verified reliable samples and identifying violations of category-level structure, capturing noisy instances that evade local detection. Third, Expert-driven Trust Evolution closes the loop by treating the noise pool not as a discard bin but as a reservoir of recoverable supervision in which a task-specific expert continuously reassesses suspicious samples as reliable evidence accumulates, progressively relabeling informative examples that were initially flagged as unreliable. Empirical results show that our approach outperforms all compared methods.

Primary area:
learning on time series and dynamical systems
Keywords:
noisy label learning, continual learning, class-incremental learning, parameter-efficient fine-tuning, LoRA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2Zn9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Progressive Trust Calibration (PTC) to address noisy-label continual learning, where label corruption and task distribution shifts interact over time. PTC initializes sample trust using neighborhood label consistency and local feature compactness in a frozen pretrained representation space. It then refines trust through global semantic verification using class prototypes. Finally, a task-specific expert dynamically reassesses suspicious samples and corrects their labels to recover useful supervision. Experiments on Split CIFAR-10, Split CIFAR-100, and CUB-200, together with ablation studies, theoretical analysis, and efficiency evaluations, support the effectiveness of the proposed method.

Critical strengths

The motivation is clearly stated and the method design is reasonable. The paper identifies the limitation that static noise filtering tends to mistakenly discard informative samples. By combining local neighborhood relations, global class prototypes and expert predictions, it develops a progressive trust calibration framework with desirable complementarity and interpretability.

Critical weaknesses

(1) The theoretical analysis lacks rigor. Some results in Appendix B rely on strong assumptions. The prototype consistency bound in Theorem B.8 and the sample recovery and convergence results in Theorems B.13–B.14 involve derivations that require further clarification. The authors should clearly state the assumptions and provide more rigorous proofs.

(2) The experimental evaluation is not comprehensive enough. Experiments with different noise rates are mainly conducted on Split-CIFAR-10, while similar evaluations on Split-CIFAR-100 and CUB-200 are missing. In addition, the paper does not provide a hyperparameter sensitivity analysis, leaving the effects of key parameters on performance and stability unclear. These limitations make it difficult to fully assess the robustness and generalization ability of the proposed method.

Questions for authors

(1) In Theorem B.8, the empirical prototypes are normalized, whereas the population class centers are defined as the expected features. Could the authors further explain why the consistency bound holds? In addition, do the relabeling error rate and convergence results in Theorems B.13–B.14 require additional assumptions, particularly regarding the conditional error on recovered samples and the stopping conditions?

(2) Equation (11) in the main text includes only the cross-entropy loss, whereas the expert training objective in Appendices C.3 and C.4 also includes a distillation term. Could the authors clarify the complete training objective used in practice, including the distillation weight and the definitions of the old and new feature representations? In addition, could the authors provide the complete optimization objective for the classifier alignment stage to improve reproducibility?

(3) Experiments with different noise rates are only conducted on Split-CIFAR-10. Could the authors supplement corresponding experiments on Split-CIFAR-100 and CUB-200? Also, how are the key hyperparameters selected?

Reviewer_pVPfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses continual learning with noisy labels in the class incremental setting, on a frozen pretrained backbone, through Progressive Trust Calibration (PTC), which treats sample reliability as a dynamic property rather than a one shot filter. PTC filters each task's data with kNN label agreement and class prototype consistency in the frozen feature space, then trains a per task LoRA expert on the trusted samples and uses it to relabel and recover flagged samples. It is evaluated on different splits of different datasets.

Critical strengths

1. Mitigating label noise in continual learning with pretrained backbones is an important direction, and the problem is well framed. The core design principle of anchoring the initial filtering in the frozen encoder's geometry, before allowing a task adapted model to relabel data, is intuitive and directly mitigates early confirmation bias.

2. Stages I and II operate entirely on cached normalized features and require no training or knowledge of the noise rate, which makes the filtering cheap and easy to deploy; as per ablations they account for most of the measured gains.

Critical weaknesses

1. Test time routing across the per task LoRA experts and the block diagonal classifier pool WW is never specified (section 3.5, Alg. 1), nor is the feature space in which μc,Σc\mu_c,\Sigma_c are estimated. Aligning WW on a single Gaussian N(μc,Σc)\mathcal{N}(\mu_c,\Sigma_c) per class either mismatches the multi-expert features seen at test time or lets heads learn expert specific feature signatures instead of class content.

2. The reported 221K parameters and 17.7 GFLOPs (Table 5) match CL-LoRA exactly, whereas a rank-8 LoRA on q,vq,v in all 12 ViT-B blocks has ≈295K parameters, suggesting an unstated shared adapter.

3. A coherent subcluster of class AA mislabeled as BB obtains high kNN agreement in Stage I (≈100% once it has at least k=20k=20 members), pulls BB's Stage II prototype toward itself, and trains the Stage III expert, which irreversibly relabels nearby clean AA samples as BB. Section 4.2 and App. B state that under Attack I flipped samples "form a compact region and are mutually nearest neighbours", but random flips do not produce such regions, so this noise type is never actually tested.

4. On Split CIFAR-10, L2P (9.30%) and SR2-LoRA\text{SR}^2\text{-LoRA} (10.01%) are at or below random guessing (10%), i.e. no better than a classifier that ignores the image. On CUB-200, DER (2.91%, Attack I) and DER++ (3.23%, Attack II) fall below the 5% reachable by a model that retains only the last task. Baseline hyperparameters, epochs and buffer sizes are not reported, so the headline margins are hard to interpret.

5. A training-free nearest class mean (NCM) baseline on the frozen ViT features is missing; it cannot memorize or forget, and within task symmetric or swap noise largely preserves its decisions, so the source of PTC's robustness is unclear.

6. PTC's "No Defense" configuration averages 64.25% on CUB (Attack I, Fig. 3a, single run) versus 77.62% for undefended CL-LoRA (Table 1), unexplained despite the apparently shared architecture; the +19-point gain of Stages I–II is thus measured from a weak reference.

7. Noise rates other than 40% are tested only on CIFAR-10 (Table 4), where both attacks coincide; clean accuracies for CIFAR-100 and CUB are missing, the noise rate never varies across tasks, and no real world noisy dataset is used.

Questions for authors
  1. (W1) How is a test prediction formed? In which feature space are μc,Σc\mu_c,\Sigma_c estimated, and how is a sampled vector fed to the block diagonal pool?
  2. (W2) Is PTC built on CL-LoRA's shared and task specific adapters?
  3. (W6) What exactly is "No Defense" in Fig. 3(a), and why is it 13 points below CL-LoRA?
  4. (W4) Please report baseline hyperparameters, epochs and buffer sizes, and explain the below chance results.

Experiments

  1. (W5) NCM on frozen features with noisy labels and with PTC cleaned labels.
  2. (W1) An ablation without the Gaussian alignment step.
  3. (W3, W7) Structured or real world noise (e.g. CIFAR-10N/100N, or flipping the feature space sub cluster nearest the partner class), and a noise rate sweep on CIFAR-100/CUB with clean results.
1 comment
AuthorsOpenReview ↗

Response 5-7: NCM controls, learner comparisons, and robustness

Q5: Frozen-feature NCM and the source of robustness.

A5: We agree that a strong frozen representation should be evaluated independently of classifier training. We therefore compare NCM using the original noisy supervision with NCM using PTC-curated samples and labels on CUB-200 at 40% label noise. Both use the same frozen ViT features and NCM decision rule. The NCM classification stage trains no parametric head and performs no Gaussian alignment.

SeedNCM: noisy supervision, Final (%)NCM: PTC-curated supervision, Final (%)Gain (points)
172.1886.47+14.29
372.2886.50+14.22
199371.7686.52+14.76
Mean ± SD72.07 ± 0.2886.50 ± 0.03+14.42 ± 0.29

SD denotes the sample standard deviation over three seeds. Gains and summary statistics are calculated before rounding. PTC improves NCM by more than 14 points in every seed. This isolates a substantial benefit from curated supervision while holding the representation and classification rule fixed; the improvement cannot be attributed solely to the pretrained features or additional classifier training. It also supports using PTC as a curation module for a simple prototype classifier. Here, training-free refers to NCM classification; obtaining PTC-curated supervision may still require expert training. The curated condition uses PTC's output, not oracle clean labels.

Q6: No Defense and the comparison with CL-LoRA.

A6: No Defense retains PTC's downstream learner, including expert training, relation regularization, classifier alignment, and the per-task training budget. It disables sample filtering and recovery and trains directly on the original noisy supervision. Expert training itself remains active. No Defense and PTC therefore form a within-learner control, whereas No Defense and CL-LoRA are different learners.

The 64.25% No Defense Avg in Fig. 3(a) is from one representative run; CL-LoRA's 77.62% Avg in Table 1 is a three-seed mean. Their 13.37-point difference is not PTC's defense gain and is not a matched single-seed comparison. The additional seed-1 controls in Answer 1 clarify the learner's behavior: without PTC, the aligned learner achieves 89.68% on clean data but 64.25% under 40% corruption; removing alignment raises the latter to 75.39%. Thus, the ablation identifies alignment as a source of vulnerability under uncurated noisy supervision, rather than attributing the original cross-method gap solely from aggregate scores.

The contribution of PTC also persists without that vulnerable component: under attack it raises Avg from 75.39% to 80.41% without alignment (+5.02 points), while the corresponding gain with alignment is 64.25% to 84.40% (+20.15 points). The new factorial control therefore separates PTC's curation benefit from the effect of Gaussian alignment. Separately, the original same-run comparison in Fig. 3(a) gives a 19.42-point Avg gain from Stages I-II (64.25% to 83.67%), and Table 2 gives Stage III gains of 2.00 Final points and 1.36 Avg points over three seeds. We have distinguished these experimental scopes and aggregation levels in the revised manuscript.

Q7: Noise-rate coverage and real-world noise.

A7: We supplement the CIFAR-100 and CUB-200 evaluation with clean training and 20% noise results. The table reports final accuracy (%) as mean ± sample SD over three runs; its 40% entries are the Attack II results already reported in Table 1.

Dataset0% (clean training)20% label noise40% label noise
CIFAR-10087.96 ± 0.1787.24 ± 0.0687.70 ± 1.06
CUB-20089.73 ± 0.0886.61 ± 0.4085.71 ± 0.55

These results extend the noise-rate evaluation beyond CIFAR-10. The complementary CIFAR-10N experiment and task-dependent cross-task contamination statistics are presented in Answer 3, where we also discuss their scope relative to a targeted coherent-subcluster attack.

We thank the reviewer again for identifying these issues. The additional NCM control, clearer inference specification, and explicit separation of learner components help us present the contribution and scope of PTC more precisely. We appreciate the opportunity to strengthen the experimental evidence and reproducibility.

Reviewer_pQMROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies label noise in class-incremental learning with a frozen pretrained ViT and per-task LoRA experts. It argues that sample reliability should evolve with the learner’s competence rather than be fixed once. PTC has three stages per task. Stage I scores each sample on frozen features by combining the fraction of its k nearest neighbors that share its label with a neighborhood-compactness score, then splits the data with Otsu’s threshold. Stage II builds class prototypes from the trusted subset and re-scores the trusted samples by the softmax probability of their observed labels over prototype similarities. Stage III trains a LoRA expert on the remaining trusted data. It then iteratively relabels samples in the noise pool to the expert’s prediction when a score combining confidence and prototype agreement exceeds an Otsu threshold. On Split CIFAR-10, CIFAR-100 and CUB-200 with 40% pairwise or symmetric corruption, PTC reports final-accuracy margins of 4.84 to 16.42 points over the strongest compared method. The appendix adds a theoretical analysis of the three stages, a corruption-ratio study and an efficiency comparison.

Critical strengths
  1. The paper addresses a practically relevant problem with a pipeline that is easy to adopt. Label corruption in continual learning with frozen pretrained backbones is a realistic setting. Estimating reliability from the frozen representation before the classifier adapts is a sensible way to avoid the early-training bias of loss- and confidence-based filters. Each stage is simple, and no noise rate needs to be specified. Table 5 reports the shortest training time among the compared methods, and lower memory use than the other LoRA-based methods. This is the paper’s main merit.

  2. The empirical study is broad and reports variability. Table 1 covers three benchmarks and two corruption types with standard deviations, and PTC ranks first in every column. The stage-wise ablation (Fig. 3 +Table 2) shows that each stage improves accuracy on CUB-200. Table 4 shows that PTC’s accuracy on Split CIFAR-10 is nearly unchanged between 0% and 40% corruption. The discussion in Section 4.2 of why local evidence is insufficient under pairwise corruption is also informative. These analyses make the paper’s claims checkable, and several of my questions build on them.

Critical weaknesses
  1. My main concern is that the experiments do not yet establish the paper’s central thesis, and the comparisons do not fully isolate the contribution of trust estimation. The abstract presents reliability as a dynamic property that evolves with the model’s semantic competence. As I understand the method, Stages I and II are computed from frozen features and observed labels and therefore do not depend on the learner. Only Stage III lets trust change with the model. In the stage-wise ablation on CUB-200 under Attack I, Stages I and II together raise the task-averaged accuracy of a single run from 64.25% to 83.67%. Stage III adds a further 2.00 points of final and 1.36 points of average accuracy over three seeds (Table 2). I could not find an ablation of Stage III in other settings, so the evidence for the component that embodies the thesis currently rests on one benchmark and one attack. Regarding the baselines, Appendix C.2 describes CL-LoRA, KeepLoRA and SR²-LoRA as noise-robust methods, but as far as I can tell none of them was designed for label noise. The main text notes that such approaches assume clean streams, Appendix D.2 states that CL-LoRA has no noise-detection mechanism, and SR²-LoRA was proposed to reduce forgetting in class-incremental learning by constraining inter-layer relation drift. The noise-robust rules (Co-teaching in Table 1, and SCE and APL in Table 4 on Split CIFAR-10) are attached to KeepLoRA, which trails CL-LoRA in every setting of Tables 1 and 4. PTC’s own learner, in turn, uses a feature-distillation term and a classifier-alignment step based on Gaussian feature resampling (Algorithm 1, Appendix C.4), which are not described in Section 3. Because the learners differ, the remaining gap may reflect differences between learners as well as trust estimation. For instance, a single run of PTC’s learner without any defense reaches 64.25% average accuracy on CUB-200 under Attack I, similar to the 65.73% of KeepLoRA with Co-teaching. Finally, Stage I’s agreement score corresponds to the k-NN label-consensus criterion of training-free detectors such as SimiFeat, which detects corrupted labels by checking label agreement among nearby features without training a model (Zhu et al., 2022). Stage II corresponds to a nearest-prototype score on the same features. A comparison with such detectors would help clarify what the progressive design adds.

  2. A second concern is that some design choices are not yet well supported, and the quality of the relabeled supervision is not measured directly. The compactness term s_i (Eqs. 4 and 5) does not depend on the observed label. Under Attack II, where labels are flipped independently of the image, it is therefore statistically independent of corruption. The gap κ in Theorem B.4 is then zero, and the term does not contribute to the mean separation established there. Since Section 3.3 notes that correctly labeled boundary samples tend to have low neighborhood similarity, the term may also lower their trust. An ablation of λ (fixed at 0.85) would help clarify its role. Stage III scores pooled samples by the expert’s confidence and prototype agreement under its own prediction (Eq. 13), and relabels recovered samples to that prediction. This can be viewed as a form of iterative self-training whose threshold is set relative to the current score distribution. The accompanying analysis relies on Assumptions B.10 and B.11, which already encode much of the intended feedback between recovery and expert improvement. Theorem B.13 establishes a vanishing relabeling error only in the realizable limit, as the authors acknowledge. Most importantly, I could not find a direct measurement of how often the labels assigned in Stage III are correct, even though the ground truth is available in this setting. The gradient-based energy in Fig. 3(c) is a useful diagnostic, but an indirect one.

Questions for authors
  1. Could the authors provide a citation or definition for the CLDP setting? Could they also clarify how the class pairs for Attack I are formed in each task, given that Fig. 3(b) reports 120 corrupted samples per CUB-200 task, and whether corrupted samples within a class are selected uniformly at random? I would also appreciate a brief explanation of the Split CIFAR-10 results of L2P and SR²-LoRA (9.30% and 10.01% final accuracy). Labels on this benchmark are flipped independently with probability 0.4, and Appendix D.4 notes that a learner retaining only its most recent task would reach about 20%. These values are therefore lower than I would expect. A short discussion, or a check of the baseline configurations, would make Table 1 easier to interpret.

  2. To isolate the contribution of trust estimation, as Appendix C.1 intends, could the authors attach Co-teaching, SCE, APL and a training-free detector such as SimiFeat (Zhu et al., 2022) to PTC’s own learner? This should include its distillation and classifier-alignment steps, with results reported on all five settings. Detection precision and recall for these detectors and for Stages I and II would also be very helpful.

  3. For each round of Stage III, could the authors report the number of recovered samples and the fraction whose assigned label is correct, separately for originally corrupted and clean samples? The label accuracy of C^(2) would also be useful. It would also help to clarify a few details that I may have misread: a.How many rounds are executed in practice, and which stopping rule is used? With E = 10 and at most five rounds (Table 3), the loop condition in Algorithm 1 appears to allow a single round; b.What does the Det ✗ configuration in Table 2 train on? Appendix E.1 states that enabling Stage III changes only the labels of the recovered samples, whereas Section 3.5 suggests that the pool is not used for training when Stage III is disabled; c.How do the negative values plotted in Fig. 3(c) relate to the energy defined on [0, 1] in Appendix E.1?

  4. Could the authors report the Stage III gain on the other four combinations of benchmark and attack? Could they also ablate λ, the per-class cap R1 and the threshold rule (Otsu’s criterion versus a fixed threshold)? Finally, it would be useful to report the contribution of the distillation term and the classifier-alignment step, which are part of PTC’s learner but are described only in the appendix.

I appreciate the practical motivation, the simplicity of the pipeline and the breadth of the reported experiments. In its current form, however, the evidence does not yet support the paper’s main claims: the component that embodies the thesis is evaluated in a single setting, where it adds about two points; the comparisons do not separate trust estimation from differences between learners; the correctness of the relabeled supervision is not measured directly.

I believe these points can be addressed. Doing so, however, would require additional experiments across all benchmarks and possibly a reframing of the central claim, which seems beyond the scope of a rebuttal. I would encourage the authors to pursue these directions and would be glad to see a revised version.

1 comment

I thank the authors for a responsive rebuttal. The protocol clarifications, the Gaussian-alignment 2×2 (Table 13), and the frozen-feature NCM control (Table 14) resolve several of my concerns. Four issues remain.

  1. The new results confirm a claim–evidence misalignment. On the shared learner (CIFAR-10, seed 1): SimiFeat 92.97 → PTC without Stage III 96.65 → full PTC 97.26. Of the 4.29-point lead over the strongest matched alternative, ~3.7 points come from static, learner-independent filtering and ~0.6 from the mechanism that operationalizes "trust evolving with competence"; Table 10 shows the same pattern (+0.5–1.4) in every setting. Even the largest estimate of the Stage III effect (+2.00 Final over three seeds, Table 2) remains smaller than the measured contributions of static curation (+5.02) or alignment (+3.99) on CUB-200 (Table 13). Table 13 further shows the alignment term is itself noise-fragile without curation (64.25 vs 75.39 attacked Avg) and carries a substantial share of the CUB-200 margins. The component that embodies the paper's central claim is the smallest measured contributor, and the claims should be reframed accordingly.

  2. The matched-learner control covers only the easiest setting. CIFAR-10 has ~5,000 images/class, where the strongest methods already reach 93–97% accuracy; CUB-200 (~30 images/class), where the authors argue recovery matters most, is untested, as is the CUB-200 Attack II cell of Table 10. Please run SimiFeat on PTC's learner on CUB-200 under both attacks, reporting detection precision and recall.

  3. Recovered-label accuracy is reported only in aggregate. A wrong relabel of a corrupted sample leaves noise in place; of a clean sample it converts correct supervision into incorrect. With 17–23% wrong recovered labels on CIFAR-100 and CUB-200, the corrupted/clean split and per-round accuracy are pure post-processing of existing logs and should be reported.

  4. The λ ablation is still missing. The revision itself now states that compactness provides no mean separation under Attack II (Sec. 3, Appendix B); λ = 0 vs 0.85 on Attack II is therefore the direct test of whether the term helps or dilutes the score. I could not find ablations of R1 or of the Otsu-versus-fixed threshold rule in the revision.

Therefore, my overall rating is unchanged, but I will raise my soundness assessment from 2 to 3, reflecting the new controls and direct measurements (Tables 10–14) that address my methodological concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.