acceptodds
Withdrawn from ICLR 2027

LiLAW: Lightweight Learnable Adaptive Weighting from Confidence and Disagreement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

High-loss training examples may be mislabeled or simply difficult to classify, making loss alone an unreliable guide to sample weighting under label noise. We introduce Lightweight Learnable Adaptive Weighting (LiLAW), which assigns sample weights using two values: the model’s confidence in the observed label and its confidence in its preferred prediction. Three global parameters control these weights and are learned by evaluating the effect of a weighted classifier update on a held-out batch, without requiring clean labels. We characterize the resulting weights and identify the conditions under which the ratio of the two probabilities distinguishes correct and incorrect annotations. Across CIFAR-10N and CIFAR-100N, LiLAW improves test accuracy in nearly all comparisons involving four frozen pretrained encoders. We also examine selective classification: abstaining on less confident predictions yields higher accuracy at lower coverage. With full-network training on Clothing1M, LiLAW achieves consistently improved results across multiple seeds, while being very lightweight. These findings suggest that a small set of learned weighting parameters can improve learning from real-world label noise and support selective classification.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
lightweight, adaptive, weighting, noise, robustness, confidence, disagreement, agreement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FUAUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In learning with label noise, the high-loss training examples can be mislabeled or correct but hard examples. The authors claim that Cross-entropy (CE), which depends only on the probability assigned to that label, cannot distinguish these hard confident examples. And cross-entropy does not describe how strongly the model favors another class. To address this, the authors propose a sample-weighting method that combines model confidence and the agreement with the observed label. The proposed method includes the ratio of the observed-class probability to the maximum probability, which can be used to reduce the influence of examples that have inconsistent observed labels with predictions. Experiments are conducted on CIFAR-N and Clothing1M to verify the proposed method.

Critical strengths
  1. The motivation of the paper is clear. Relying only on cross-entropy loss cannot hard confident examples and examples with incorrect labels. The authors consider both the model’s confidence in the observed label and its confidence in its preferred prediction.
  2. The proposed method is simple and straightforward.
  3. The authors conduct experiments on CIFAR-N and Clothing1M to verify the proposed method.
  4. The authors provide visualizations of easy and hard examples in the appendix, which can enable readers to understand the easy and hard examples intuitively.
Critical weaknesses
  1. The authors do not provide ablation studies to verify the effectiveness of key components, such as the three weightings and the ratio p_i/q_i.
  2. The research in this paper is directly related to the research direction for hard confident examples. But the manuscript neither sufficiently discusses this direction nor includes relevant papers that study hard confident examples.
  3. Experiments on CIFAR-N only include the baseline CE, and other baselines are missing.
Questions for authors

Please see the critical weaknesses.

Reviewer_onpBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Paper combats label noise via loss modification based on a 3-part taxonomy which revolves around the (maximal) confidence of the machine and what it says about the potentially noisy label. There are 3 global parameters affecting the loss modification and they are learned on a noisy validation set and the overhead the proposition incurs is rather low compared to many other propositions in the literature. Experiment on Clothing1M shows it is also performant compared to such prior works. Method is also employed in selective classification to show its applicability.

Critical strengths
  1. Proposition is easy to implement: a loss modification technology where weights are driven based on the machine's softmaxed predictions through an interplay between the maximum output and the output assigned to the observed label's slot.
    • why it is consequential to your overall assessment: many of the prior work are about complicated pipelines.
Critical weaknesses
  1. How the loss is modified is not well-justified in my opinion. For example, the β\beta component is targeting the "random" annotation errors, i.e., the machine is very highly confident that the observed label is wrong. Firstly, these are rather rare occurences in practice (misclicks, communication issues etc.) and more importantly, the network should not be punished for these instances; that's what the vanilla CE does: it is oblivious to this fact and wants the network to always fit to the noise there. The WβW_\beta being high is causing a similar effect, which seems unnecessary/harmful. Another point is the lack of discussion on KK-dependence on these α\alpha et al., i.e., number of classes seems like an important aspect for their effects. As an example, let K=2K = 2, pi=qi=0.5p_i = q_i = 0.5 and α=10\alpha = 10. These yield Wα=σ(4.5)≈0.99W_\alpha = \sigma(4.5) \approx 0.99, which is very high: 1st taxonomy in Line 155 implies there is agreement with the observed label but actually there isn't in this case; in fact, it has the highest entropy possible. So one at least expects a mention of KK-dependence of α\alpha (and a closer look for the other 2 would perhaps unveil a similar case). Lastly, the order enforcements on α\alpha et al. (4th step in Algorithm 1) is mysterious: why and how of it are left unexplained (why the need for it, and how is it enforced during updates).

    • why it is consequential to your overall assessment: the main driving part of the contribution has questionable aspects
  2. While the Clothing1M experiment compares against other methods from the literature, CIFAR-based ones do not, which is unclear why. Also, the authors use different pretrained machines in a frozen way there: what is the motivation behind this? The text does not mention a rationale. The gains over vanilla CE seem low (mostly <1%) with these frozen encoders; normally (i.e., not with frozen encoers) one has 8-10% gain in absolute terms (e.g., see Table 4 in "Robust Training under Label Noise by Over-parameterization" by Liu et al. in ICML 2022); how could this be explained for, say, R18 and 50 (since the other two already achieve high scores)? Speaking of which, when some of those encoders reach quite high accuracies already, does one need to act against label noise? Isn't this shadowing the contribution of the proposed method? Lastly, for Clothing1M, Lines 367-368 note that "Although there exist other baselines with slightly higher accuracies, most or all of them require a significantly higher time complexity and/or space complexity compared to LILAW": Firstly, I think a citation or two is needed here for those works. Secondly, there is 1 important work here not in this category: "Learning with Feature-Dependent Label Noise: A Progressive Approach" by Zhang et al. in ICLR 2021. Firstly, their algorithm is also based on the tension between maximum probability and probability on the observed label like LILAW, perhaps worth citing and comparing because of this. Secondly, Algorithm 1 seems very lightweight to implement with a neural network, they report 74.02% and they actually don't even have extra parameters to learn like α\alpha et al.

    • why it is consequential to your overall assessment: some experiments are not well-motivated I think and not favorable to the proposed method, and also an important very related work is missing in Related Work and in Experiments that undermines some claims made.
Questions for authors
  • Lines 203-204: "Multiplication by pi/qi then reduces the contribution of strongly contradicted labels.": Isn't this undermining β\beta's design? Or shouldn't β\beta et al. be able to deal with this already?

  • Lines 367-368: "Although there exist other baselines with slightly higher accuracies, most or all of them require a significantly higher time complexity and/or space complexity compared to LILAW": Could the authors provide citations of such methods? Otherwise the claim goes unsupported.

Reviewer_7VtmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learning with noisy labels. The authors propose a per-sample weighting of the loss (LiLAW). The weighting is a function of the probability of the observed label p and the maximum probability q, with three global scalars that are learned by looking ahead one-step on the (noisy) validation batch. The authors evaluate their approach on CIFAR-N with frozen features against Cross-Entropy loss only, on Clothing1M, and for selective classification.

Critical strengths

(1) The paper proposes a simple weighting that is cheap and works with noisy labels.

(2) The paper evaluates on CIFAR-N with real annotations, five seeds which supports part of claim.

Critical weaknesses

(1) The paper does not run a controlled experiment on Clothing1M. It is not clear how much of the gain in a real-world setting is due to the weighting scheme. The paper reports Cross-Entropy (CE) as 69.21, taken from the literature. Moreover, six rows of Table 2 (APNL, Mixup, Dyn-Mixup, EG, CIW, CICW) match numbers reported by Kumar and Amid [15], who have tuned CE to 70.32 (hence reducing the gain of the proposed approach to +2.18 from +3.29). In addition, there are several differences in recipe/sampling/checkpoint selection (e.g., [15] fine-tunes for 8 epochs, whereas the proposed approach trains for 80).

(2) The paper misses several comparisons to the prior work. The paper does not consider, for example, L2B (arXiv:2202.04291) or simple bootstrapping. The proposed approach shares the principle of looking ahead with L2RW and MW-Net. On CIFAR-N, the only comparison is with CE. It is difficult to assess the novelty of the paper or how "lightweight" it is.

(3) The paper does not include an ablation study. From Eqs 3-8, it looks like the weight is approximately p/q. It might be possible to match the proposed approach by setting p/q by hand.

Questions for authors

(1) Can the authors re-run CE for Clothing1M under the proposed recipe? Where do the rows of Table 2 come from?

(2) Why is the gain negative (−0.54 and −0.35) for ResNet-50 on Aggregate and Random 1?

Please compare head-to-head against L2RW, MW-Net, L2B, and/or GCE/SCE under the same protocol.

Please include an ablation study for the proposed weighting. In particular, please control for setting p/q by hand. What are the learned values of \alpha, \beta, \delta?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.