acceptodds
Under review as a conference paper at ICLR 2027

Towards Bregman Loss with Aitchison-Inspired Potential for Label Distribution Learning

Abstract

Label Distribution Learning (LDL) represents label ambiguity using simplex-valued vectors over the label space, referred to as label distributions. In most applications, the observed training label distribution is aggregated from a finite number of annotations and can, under an unbiased sampling mechanism, be viewed as a finite-sample estimate of an underlying population label distribution rather than an error-free target. Under this assumption, the conditional arithmetic mean of label distributions equals the underlying population label distribution. Consequently, an LDL point prediction should approximate the conditional arithmetic mean, rather than an individual realization. This motivates strict arithmetic-mean properness of the training loss, requiring the conditional arithmetic mean to be its unique conditional Bayes prediction. Although the widely used Kullback-Leibler divergence and squared Euclidean loss satisfy this property, under restricted model capacity or data noise, small errors in probability space can still induce substantial distortions in relative label ratios, particularly near the simplex boundary. Such distortions can be detrimental when relative label proportions are critical to downstream decisions. We therefore investigate how to reconcile mean-properness with ratio-sensitivity in LDL. Since the squared Aitchison loss directly captures relative variation through log-ratios but is generally not arithmetic-mean proper, we seek a mean-proper loss with Aitchison local geometry as its reference. Specifically, we propose a Bregman loss with Aitchison-inspired potential, which is derived by matching the coordinatewise curvature scaling of its potential function to local Aitchison geometry. We establish its mean-properness, quantitatively compare its local geometry with Aitchison, and show its merits over conventional LDL losses. Finally, we conduct extensive experiments to show the effectiveness of our method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Label Distribution Learning, Mean Properness, Relative Label Ratio

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_RAwWOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates LDL loss functions from the perspectives of mean-properness and local geometry. Mean-properness concerns the statistical target associated with the conditional expected risk of the loss function, while the local geometry focuses on how the loss function locally weights prediction errors in different directions. The authors note that while KL and SE losses exhibit strict mean-properness, they fail to provide direct and uniform control over label ratios, particular in the cases where label mass approaches zero; conversely, the squared Aitchison loss directly measures label ratio error but typically lacks mean-properness. To address this, the paper proposes a Bregman loss with Aitchison-inspired potential function, named BAP. It establishes the strict mean-properness, while also theoretically simulates the local geometry of Aitchison error. Experimental evaluations across eight datasets demonstrate the effectiveness of the proposal.

Critical strengths
  1. This paper’s research problem is significant and clearly defined. The paper analyzes LDL loss functions from the dual perspectives of statistical target and local loss geometry, clearly distinguishing between ‘‘what is being predicted” and “how local prediction errors are weighted”, thereby offering a valuable perspective for LDL loss design. Building on this analytical framework, the authors link two types of information that are practically significant yet previously unconsidered in tandem: the conditional arithmetic mean target (which is essential in the practical scenarios of finite annotations) and inter-label ratio information (which is essential in certain important applications such as Breast-cancer tumour-microenvironment deconvolution).

  2. The proposed BAP is concise yet effective, and its key properties are supported by a solid theoretical foundation. By employing a smooth quartic extension to handle the singularity of the negative logarithmic potential at zero, BAP accommodates label distributions with zero components in label ratio information mining, which is of practical significance. Simultaneously, BAP preserves the curvature of negative-log potential function above the threshold to approximate the Aitchison error structure. The paper establishes the local relationship between BAP and Aitchison error, as well as its strict mean-properness.

  3. The experiments align closely with the paper's focus on limited annotation and label-ratio prediction. By comparing LDL models across eight datasets under two annotation budgets, the authors demonstrate consistent improvements in BAP regarding the ratio error metric MALR, as well as the standard LDL metrics. Furthermore, applying BAP to several SOTA LDL algorithms reinforces its practical value as a plug-in component.

Critical weaknesses
  1. Practical recommendations for threshold selection could be clearer. While the paper provides a comprehensive sensitivity analysis, it lacks a summary of these results into concise, actionable guidelines. Providing a recommended initial threshold or a selection procedure would enhance the method's usability.

  2. There are a few issues regarding notation and typesetting. For instance, Table 1 uses (n_b) while the main text uses (n_a); additionally, the expanded summation in Equation (9) requires parentheses. It is recommended to review these details for consistency to further enhance the paper's readability.

Questions for authors

Based on the existing threshold sensitivity analysis, do the authors recommend using (\tau=0.05) as an initial setting for practical applications, followed by adjustment using a validation set?

Reviewer_USTwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigated how to design a mean-proper loss function with ratio-sensitivity for label distribution learning. The authors showed that KL divergence and SE loss possess arithmetic-mean properness but do not provide uniform control of log-ratio distortion near the simplex boundary, while squared Aitchison loss directly captures relative variation but generally does not preserve the conditional arithmetic mean. They proposed a Bregman divergence with an Aitchison-inspired potential, which may be useful for the community.

Critical strengths
  1. The paper decomposes the design of loss functions into two distinct issues: what population quantity is being elicited (the statistical objective) and how different errors are weighted (local geometry). It precisely situates three commonly used loss functions within this framework, providing a well-reasoned justification for its approach.
  2. The paper reformulates the design of mean-proper losses as the design of Bregman potential functions: Proposition 3.3 provides a Bregman representation for regular mean-proper losses, while Proposition 3.6 establishes the strict arithmetic-mean properness of BAP and derives a lower bound on the excess risk. These theoretical results provide support for the paper.
  3. The paper compares KL, SE, SA, KL+SA, and SE+SA across eight LDL datasets, and integrates BAP into three recent methods, such as Delta-LDL, DVS-LDL, and RKNN-LDL, to validate its potential as a general-purpose fitting loss.
Critical weaknesses
  1. The ratio sensitivity of BAP is merely a local, coordinate-level approximation; the paper does not provide an explicit upper bound for the error in pairwise log-ratios, making it difficult to theoretically demonstrate that it necessarily controls MALR better than KL/SE.
  2. The main experiments employ a linear model with L-BFGS; however, extensions to non-linear models, particularly deep networks, are absent.
Questions for authors
  1. Why is the curvature index set to 2? Could you provide the theoretical basis for choosing α=2?
  2. Can this proposed loss function be used in deep neural networks? Are there any particular costs or trade-offs involved?
Reviewer_mf9VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors focus on loss design for label distribution learning (LDL). Typical losses such as KL divergence and squared loss may overlook errors in relative label ratios, while squared Aitchison loss generally does not recover the conditional mean/true label distribution. To address this issue, a Bregman loss with Aitchison-inspired potential (BAP) is further constructed. The proposed loss preserves strict arithmetic-mean properness and conducts importance reweighting on errors in small label masses, while remaining finite at zero label masses. Experiments on eight LDL datasets show the effectiveness of this method, including its use in three existing LDL algorithms.

Critical strengths
  1. The proposed Aitchison-inspired potential gives a strictly arithmetic-mean proper Bregman loss (to be more detailed, it is a proper loss w.r.t. the class probability), addressing the target mismatch of squared Aitchison loss.

  2. The consideration of relative label ratios is reasonable for LDL. Besides eliciting the conditional arithmetic mean as a linear property, the loss design further considers errors in relative label proportions, which may be important even for labels with small masses.

  3. The experiments include eight LDL datasets and two annotation counts. Replacing the fitting losses of three existing LDL algorithms with BAP further shows its effectiveness, particularly in reducing log-ratio errors.

Critical weaknesses

While this work focuses on LDL and provides an interesting potential design based on relative label ratios, its connection with proper losses and property elicitation could be further discussed. In particular, the target π(x)=ED∼Px[D]\pi(x)=\mathbb E_{D\sim P_x}[D] is a linear property of the conditional distribution PxP_x. Although the connection with [1] is already discussed in Section 3.2, I suggest further including works on multiclass proper losses [2], vector-valued property elicitation [3], and other potential-based loss functions [4,5,6], and discussing their relation to the proposed construction for a more complete literature review.

[1] Jacob D. Abernethy, Rafael M. Frongillo. A Characterization of Scoring Rules for Linear Properties. COLT, 2012.

[2] Robert C. Williamson, Elodie Vernet, Mark D. Reid. Composite Multiclass Losses. Journal of Machine Learning Research, 17(222):1–52, 2016.

[3] Rafael Frongillo, Ian A. Kash. Vector-Valued Property Elicitation. COLT, 2015.

[4] Mathieu Blondel, André F. T. Martins, Vlad Niculae. Learning with Fenchel-Young Losses. Journal of Machine Learning Research, 21(35):1–69, 2020.

[5] Mathieu Blondel. Structured Prediction with Projection Oracles. NeurIPS, 2019.

[6] Yuzhou Cao, Han Bao, Lei Feng, Bo An. Establishing Linear Surrogate Regret Bounds for Convex Smooth Losses via Convolutional Fenchel–Young Losses. NeurIPS, 2025.

Questions for authors

Please see the weaknesses.

Comments on the paper

1 comment

Dear Reviewers,

Thanks for your time and effort in reviewing this paper. The authors have responded to the comments by providing a rebuttal. Please read the rebuttal and indicate whether your concerns have been resolved. Thanks.

Regards from the AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.