acceptodds
Under review as a conference paper at ICLR 2027

A Blockwise Probability Loss Function for Multiclass Classification Learning

Abstract

Binary loss functions are very popular in multiclass classification learning algorithms due to their low computational complexity. On the other hand, some of them, such as softmax cross-entropy (CE) and binary cross-entropy (BCE) may impose restrictions on class separability or algorithm convergence. In standard multiclass classification, cross‑entropy is typically used with one‑hot targets, while the BCE reduces to the case where all outputs of the learned function are treated separately. We observe that if binary losses consider the whole output as a collection of separate binary outputs, they do not fully explore the extrinsic information from other components during training. Motivated by that, we present a new blockwise loss function that satisfies three basic properties: it is (i) not separable into binary subproblems, (ii) well defined for associating classes with binary vectors, which we refer to as binary code assignments, and (iii) obtained by comparing the output to the true class representation as well as to all other classes. We show that the proposed loss reduces to CE when one-hot coding is applied, while it reduces to a scaled version of BCE when property (iii) is simplified such that the loss compares the output only with the true class representation. Furthermore, when one-hot coding is replaced with error correcting output codes (ECOCs), the proposed loss achieves accuracy that is equal to or higher than that of classical bitwise losses. Experiments were conducted on the CIFAR-100 and Tiny ImageNet datasets for various losses from the literature and demonstrate the advantages of the new blockwise probability (BP) loss in combination with ECOCs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
error correction output codes, softmax cross-entropy, loss functions, one-hot coding, multiclass classification, binary cross-entropy, cosine similarity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7wezOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the losses used during model training and inference (decoding in the paper), introduces bitwise and blockwise training losses, and proposes a blockwise probability training loss that can be applied to arbitrary binary codebooks while incorporating both general cross-entropy (CE) loss and binary cross-entropy (BCE) loss within its framework.

Critical strengths

The paper proposes a framework named blockwise probability loss that incorporates the commonly used cross-entropy (CE) loss and binary cross-entropy (BCE) loss, and it also accommodates the MSE loss and cosine similarity loss.

Critical weaknesses
  1. The issue discussed in this paper does not appear to be one that urgently needs to be addressed, has wide-ranging impact, or has attracted broad attention.
  2. The proposed BP loss is obviously more complex than the commonly used CE or BCE loss.
  3. The experimental results in Tables 1 and 2 do not provide strong evidence of the superiority of BP loss compared to CE or BCE loss. Furthermore, training ResNet34 with CE loss on CIFAR-100 typically achieves test accuracy exceeding 70%, whereas the accuracy reported in the paper is merely 57.04%, which is difficult to find convincing.
Questions for authors

What are logit-normalized versions of different loss functions?

Reviewer_CPsvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors focus on the training loss for multiclass classification with error-correcting output codes (ECOCs). Typical bitwise losses consider each output separately during training, which may overlook the comparison between the true class codeword and other candidates. A blockwise probability (BP) loss is further constructed by normalizing the products of bit probabilities over all class codewords, which allows the prediction system to compare these candidates jointly during training. Experiments on CIFAR-100 and Tiny ImageNet show that BP with logit normalization provides better accuracy than the tested bitwise losses with non-one-hot codebooks.

Critical strengths
  1. The proposed loss considers other class codewords during training, compared with fitting each bit separately. I think this is reasonable for distinguishing classes whose codewords are close to each other.

  2. The experiments include different codebooks and two network architectures. In particular, the results with ResNet34 show that an accuracy close to the one-hot case can be obtained with fewer outputs.

Critical weaknesses

I noticed that the Bayes consistency/classification-calibration of the proposed BP loss is not analyzed, and I do find a case where the proposed BP loss is not calibrated with the 0-1 loss. For example, given a sample with class probability η=[0.40;0.05;0.30;0.25]\eta=[0.40;0.05;0.30;0.25] (ηm=P(Y=m∣X=x)\eta_m=\mathbb P(Y=m\mid X=x)) and the codebook C=[−1,−1;−1,+1;+1,−1;+1,+1]C=[-1,-1;-1,+1;+1,-1;+1,+1]. Then

= \ln(1+e^{z_1})+\ln(1+e^{z_2}).$$ $$L_{\mathrm{BP}}(2,z,C) = \ln(1+e^{z_1})+\ln(1+e^{-z_2}).$$ $$L_{\mathrm{BP}}(3,z,C) = \ln(1+e^{-z_1})+\ln(1+e^{z_2}).$$ $$L_{\mathrm{BP}}(4,z,C) = \ln(1+e^{-z_1})+\ln(1+e^{-z_2}).$$ The expectation of this loss \$ \log(1+e^{z_1}) + \log(1+e^{z_2}) - 0.55z_1 - 0.30z_2\$, which is minimized at $z_1=\log(11/9),z_2=\log(3/7)$. Finally the decoded prediction is $y=3$, which does not correspond to the most probable class $y=1$.
Questions for authors

Please see the weaknesses section. I will adjust the rating if these issues are solved.

Reviewer_nLbjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a blockwise probability (BP) loss for multiclass classification with error-correcting output codes. The loss jointly considers all class codewords rather than optimizing individual outputs separately. The authors establish connections to CE and BCE and evaluate the method with different codebook designs on CIFAR-100 and Tiny ImageNet.

Critical strengths
  1. The paper addresses a relevant question about training classifiers with non-one-hot class representations.
  2. The proposed formulation provides explicit connections to CE and BCE.
  3. The experiments examine multiple codebook designs, including random, compressed, and degenerate configurations.
Critical weaknesses
  1. The novelty beyond standard CE is unclear. BP is equivalent to standard CE after a fixed codebook projection, not only in the one-hot case. The contribution beyond this reparameterization needs clarification.
  2. The practical advantage over traditional CE classification is not consistently demonstrated. For example, on ResNet34/CIFAR-100, normalized BP with a random N=100 codebook achieves 56.43%, compared with 56.57% for standard CE. Comparisons with fixed-prototype softmax classifiers and matched logit-scaling controls would help clarify the benefits.
  3. The experimental scope is limited to two low-resolution datasets, CIFAR-100 and Tiny ImageNet, using ResNet34 and a small custom CNN. For a broadly applicable classification loss, this provides insufficient evidence of generality. Larger-scale datasets, more classes, and different model families would strengthen the evaluation.
  4. The error-correction evaluation does not directly measure successful correction. Appendix a3 counts bit changes relative to the decoded codeword, which may correspond to an incorrect class. More flipped bits therefore do not necessarily indicate stronger error correction. Controlled perturbation experiments measuring correct decoding would be more informative.
  5. The comparison appears sensitive to loss scale and optimization. Although eq. (23) establishes Lbit,BP=NLBCEL_{\mathrm{bit,BP}}=N L_{\mathrm{BCE}}, their one-hot accuracies on ResNet34/Tiny ImageNet differ substantially: 48.50% versus 15.94%. These discrepancies require explanation, alongside matched tuning budgets and standard deviations rather than only mean/max results.
Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.